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ABSTRACT 


This  report  provides  an  overview  of  performance  assurance  practices 
developed  and  applied  since  World  War  II  to  improve  the  cost-effectiveness 
of  selected,  large-scale  technological  systems.  The  purpose  is  to  present 
concepts  which  might  be  applicable  in  reducing  the  frequency  and  duration 
of  electric  powerplant  outages.  A comprehensive  performance  assurance 
program  is  recommended.  The  elements  of  the  suggested  program  were  derived 
by  considering  elements  of  precedent  programs  which  are  believed  to  be  most 
effective  in  other  applications. 


iii 


CONTENTS 


Page 

ABSTRACT iii 


CHAPTER  ONE:  INTRODUCTION i-1 

1.1  Background 1-1 

1.2  Rationale  for  the  ARINC  Research  Study  1-1 

1.3  Study  Objectives  1-2 

1.4  Study  Approach 

1.5  Scope  of  the  Study 

1.5.1  Individuals  Interviewed  1-3 

1.5.2  Content  of  the  Interviews 1-3 

1.5.3  Performance  Assurance  Programs  Selected  for 

Review  and  Analysis 1-5 

1.5.4  Key  Documents 1-6 

1.6  Guide  to  This  Report 1-6 

CHAPTER  TWO:  RESULTS  AND  RECOMMENDATIONS  2-1 

2.1  Results  of  the  Survey 2-1 

2.1.1  The  Performance  Assurance  Function  2-1 

2.1.2  Implementing  Performance  Assurance  2-1 

2.1.3  Delegation  of  Effort 2-3 

2.1.4  The  Evolution  of  Performance  Assurance  Programs  . . 2-3 

2.1.5  Specifying  Performance  Goals  ....  2-6 

2.1.6  Specifying  Redundancy  2-7 

2.1.7  Specifying  Operating  Profiles,  Levels  of  Stress, 

and  Derating  Requirements  2-7 

2.1.8  Specifying  Rates  of  Technological  Change  2-7 

2.1.9  Requiring  a Performance  Assurance  Program  Plan  . . . 2-7 

2.1.10  Cost  and  Effectiveness  of  Contractual  Performance 

Assurance 2-7 

2.2  Recommendations 2-9 

APPENDIX  A:  PERFORMANCE  ASSURANCE  AS  PRACTICED  BY  THE  U.S. 

DEPARTMENT  OF  DEFENSE  A-l 

APPENDIX  B:  PERFORMANCE  ASSURANCE  AS  PRACTICED  BY  THE  NATIONAL 

AERONAUTICS  AND  SPACE  ADMINISTRATION  B-l 


£ 


l\ 

r "a 

1 ' 

CONTENTS  (continued) 

r *■- 

l < 

v 

l 

Page 

I >- 

APPENDIX 

C: 

PERFORMANCE  ASSURANCE  AS  PRACTICED  BY  THE  FEDERAL 
AVIATION  ADMINISTRATION  AND  AIRLINE  INDUSTRY  .... 

. . C-l 

[ 

l « 

APPENDIX 

D: 

PERFORMANCE  ASSURANCE  PRACTICES  OF  THE  NATIONAL 
TRANSPORTATION  SAFETY  BOARD  

. . D-l 

1 *' 

APPENDIX 

E: 

PERFORMANCE  ASSURANCE  PRACTICES  OF  THE  URBAN  MASS 
TRANSIT  ADMINISTRATION  (UMTA)  

. . E-l 

APPENDIX 

F: 

PERFORMANCE  ASSURANCE  AS  PRACTICED  BY  THE  NUCLEAR 
REGULATORY  COMMISSION  (NRC)  

. . F-l 

APPENDIX 

G: 

AMERICAN  TELEPHONE  AND  TELEGRAPH  (AT&T)  

• • G— 1 

CHAPTER  ONE 


INTRODUCTION 


1.1  BACKGROUND 

A work  plan  for  the  Federal  Electric  Utility  Program  Task  Force  was 
set  forth  in  a joint  FEA-ERDA  memorandum  dated  July  1,  1977.*  A principal 
activity  described  was  the  development  of  a Department  of  Energy  (DoE) 
program  for  improving  baseload  powerplant  performance.  To  carry  out  the 
activity,  a Powerplant  Performance  Working  Group  was  organized  with  rep- 
resentatives of  the  Federal  Energy  Administration  (FEA) , the  Energy  Research 
Development  Administration  (ERDA) , and  the  Federal  Power  Commission  (FPC) . 

The  Working  Group  was  activated  on  August  19,  1977,  and  assigned  to 
conduct  a study  and  submit  (1)  a recommendation  for  a comprehensive  DoE 
program  and  (2)  an  assessment  of  the  potential  benefits  to  be  derived  from 
improved  powerplant  performance.  The  study  was  to  identify  program  re- 
quirements for  legislation,  research,  development,  demonstration,  technical 
assistance,  and  standards  development,  as  well  as  the  DoE  resources  needed 
for  implementation. 

Under  Purchase  Order  No.  EX-77-X-01-4120,  dated  September  9,  1977,  the 
executive  steering  committee  requested  ARINC  Research  Corporation  to  perform 
a complementary  study  in  parallel  with  the  Working  Group's  efforts. 


1.2  RATIONALE  FOR  THE  ARINC  RESEARCH  STUDY 

Electric  utilities  must  have  generating  capability  to  meet  peak  demand; 
in  consequence,  a part  of  their  plant  is  idle  when  demand  is  less  them  peak. 
The  need  to  reduce  this  idle  baseload  capacity  grows  in  importance  as 
pressures  increase  to  hold  the  line  on  costs,  as  siting  and  licensing  con- 
straints make  it  more  difficult  to  add  new  capacity,  and  as  the  nation 
attempts  to  conserve  scarce  energy  resources. 


♦Memorandum  to  Jack  O'Leary  (Administrator,  FEA)  and  Robert  Fri  (Acting 
Administrator,  ERDA)  prepared  by  William  Fischer  (Associate  Administrator 
for  Policy  and  Program  Analysis,  FEA)  and  David  Israel  (Assistant 
Administrator  for  Field  Operations,  ERDA). 
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One  approach  is  to  implement  a formal  performance  assurance  program 
aimed  at  reducing  the  frequency  and  duration  of  powerplant  outages.  If  such 
a program  could  be  applied  successfully,  it  would  be  possible  for  less 
baseload  capacity  to  provide  adequate  reserve  margins. 

The  problem  of  achieving  the  maximum  reduction  in  the  frequency  and 
duration  of  both  forced  and  scheduled  outages  is  equivalent  to  that  of  re- 
ducing the  failure  rate,  repair  time,  and  preventive  maintenance  time  asso- 
ciated with  each  critical  component  as  well  as  the  system  as  a whole.  This 
problem  is  typically  addressed  in  the  normal  course  of  applying  traditional 
procurement  and  engineering  practices.  However,  as  technology  changes  and 
systems  (i.e.,  powerplants)  become  larger  and  more  complex,  it  may  be  cost- 
effective  to  augment  traditional  procurement  and  engineering  practices  in 
such  a way  as  to  focus  more  emphasis  on  plant  reliability,  maintainability, 
and  quality  assurance.  Furthermore,  it  may  be  appropriate  to  organize 
and  share  the  effort  on  a broader- than- local  scale. 


1 . 3 STUDY  OBJECTIVES 

The  objectives  of  the  ARINC  Research  study  were  to  review  the  substance 
of  performance  assurance  programs  applied  by  selected  government  agencies 
and  industries  and  to  derive  from  experience  in  such  agencies  and  industries 
a comprehensive  performance  assurance  program  to  be  considered  for 
application  in  the  electric  power  industry. 

1 . 4 STUDY  APPROACH 

The  approach  was  to  interview  persons  responsible  for  developing  or 
applying  performance  assurance  programs  in  selected  agencies  and  companies 
and  to  review  and  analyze  program  documents  identified  during  these 
interviews.  Program  elements  identified  by  interviewees  as  being  of  sub- 
stantial value  in  the  context  of  their  own  experience  were  then  combined 
and  fitted  together  to  form  a comprehensive,  recommended  program  for  ap- 
plication in  the  electric  power  industry.  ARINC  Research  Corporation  made 
the  final  judgments  concerning  which  program  elements  to  include  in  the 
recommended  program.  However,  in  making  these  judgments,  we  tried  to 
consider  and  incorporate  a spectrum  of  other  perspectives.  We  found  the 
following  proceedings  and  reports  to  be  most  valuable  in  understanding 
these  other  perspectives: 

• "Proceedings  of  the  Reliability  Engineering  Conference  for  the 
Electric  Power  Industry",  February  1974. 

"A  Report  on  Improving  the  Productivity  of  Electric  Powerplants", 

FEA  Interagency  Task  Group  on  Powerplant  Reliability,  March  1975. 

"Electrical  Generating  Plant  Availability" , FPC  Bureau  of  Power 
Staff  Report,  May  1975. 

"Requirements  for  Reliability  Analysis  in  the  Design  and  Operation 
of  Safety  Systems  for  Nuclear  Power  Generating  Stations",  IEEE 
Draft  Standard  P577/02/REV2 , April  1976. 
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"Proceedings  of  the  Executive  Conference  on  Improving  Power  Plant 
Reliability",  September  1976. 

"Use  of  Nuclear  Plant  Operating  Experience  to  Guide  Productivity 
Improvement  Programs",  prepared  by  M.E . Lapides  and  E.  Zebroski, 
Electric  Power  Research  Institute,  November  1976. 

"Draft  Guidelines  for  Incorporating  Performance  Assurance  Require- 
ments in  Fossil  Energy  Research,  Development  and  Demonstration 
Contracts",  Assistant  Administrator  for  Fossil  Energy,  Energy 
Research  and  Development  Administration  (ERDA) , May  20,  1977. 

"Proceedings  of  the  Fourth  Reliability  Engineering  Conference  for 
the  Electric  Power  Industry",  June  1977. 

"Availability  of  Fossil-Fired  Steam  Power  Plants" , Special  Report 
FP-422-SR,  prepared  by  Don  Anson,  Electric  Power  Research  Institute, 
June  1977. 


1.5  SCOPE  OF  THE  STUDY 
1.5.1  Individuals  Interviewed 


More  than  fifty  persons  were  interviewed  by  telephone  to  identify 
individuals  with  detailed  knowledge  of  performance  assurance  programs  and 
practices  and  obtain  information  necessary  to  focus  the  effort.  Of  the 
individuals  so  identified,  those  located  in  the  Washington  - Annapolis  area 
were  interviewed  face-to-face;  they  are  identified  in  Table  1-1. 

1.5.2  Content  of  the  Interviews 


The  interviews  were  structured  around  questions  relating  to  the  scope 
and  specific  content,  background,  age,  management,  and  organization  of 
performance  assurance  programs  in  which  the  interviewee  had  a major  role. 
Most  persons  interviewed  had  many  years  of  experience  in  developing  and 
managing  formal  performance  assurance  programs  and  could  therefore  offer 
valuable  insights  concerning  the  key  elements,  cost,  and  effectiveness  of 
formalizing  performance  assurance. 

The  principal  emphasis  during  the  interviews  was  to  identify  and  con- 
centrate on  those  elements  or  aspects  of  the  program  which  interviewees 
believed  to  be  most  cost-effective.  These  elements  and  aspects  were 
identified  by  asking  the  following  questions: 

• Which  elements  or  aspects  have  you  found  most  cost-effective  in: 

• • Detecting  and  correcting  failure  causes  and  mechanisms  during 
pre-operational  phases? 

••  Reducing  maintenance  costs? 

• • Reducing  repair  and  replacement  costs? 

• • Reducing  redundancy  requirements? 
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Table  1-1.  INDIVIDUALS  WHO  CONTRIBUTED  TO  THE  STUDY  IN  FACE-TO-FACE 
INTERVIEWS 


M.  Barrett 
Staff  Supervisor 
AT&T  Long  Lines 

E.  J.  Boyle 

Chief,  System  Safety  Division 
Urban  Mass  Transit  Administration 

H.  Frankel 

Assistant  Director 

Materials  and  Power  Generation 

Fossil  Energy 

U.S.  Department  of  Energy 

J.  J.  Genovese 

Assistant  Deputy  Chief  of  Naval 
Material,  Logistics 
Naval  Material  Command 

C.  J.  Heltemes,  Jr. 

Chief,  Quality  Assurance  Branch 
Division  of  Project  Management 
U.S.  Nuclear  Regulatory  Commission 

A.  Lakner 

Chief,  Reliability  and  Maintainability 
Systems  Engineering 
Federal  Aviation  Administration 

A.  J.  Moskovitz 

Chief,  Office  of  Space  Science 
Program  Assurance 
NASA  Headquarters 


F.  Newhouse 

ARINC  Service  Manager 

AT&T  Long  Lines 

T.  Palmer 
District  Engineer 
C&P  Telephone  Company 

(Toll  Terminals) 

H.  D.  Short 

ARINC  National  Account  Manager 
AT&T  Long  Lines 

F.  Starbuck 

Assistant  Director  for  Facilities 
and  Equipment 
Engineering  Directorate 
Goddard  Space  Flight  Center 

Colonel  B.  H.  Swett 
Military  Assistant  to  the 

Assistant  Director  of  Planning 

U. S.  Department  of  Defense,  ODDR&E 

F.  T.  Taylor 

Director,  Bureau  of  Accident 
Investigation 

National  Transportation  Safety  Board 

W.  j.  Willoughby,  Jr. 

Assistant  Deputy  Chief  of  Naval 

Material,  Reliability  & Engineering 
Naval  Material  Command 


••  Reducing  acquisition  costs? 

• • Reducing  performance  assurance  costs? 

••  Reducing  program  support  costs? 

• Which  elements  or  aspects  of  your  program  are  most  enthusiastically 
accepted  by: 

• • General  management? 

• • Engineering  management? 

• • Government? 

••  Contractors  and  suppliers? 

••  Customers,  consumers,  users? 

• Which  elements  or  aspects  of  your  program  were  derived  from  experience 
in  other  agencies  or  industries,  i.e.,  which  are  commonly  applicable 
in  a broad  spectrum  of  technological  situations? 
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• Are  you  changing  any  element  or  aspect  of  your  program,  and 

• • Why? 

• * How? 

• Which  elements  or  aspects  of  your  program  could  be  applied  most 

effectively  if  you  were  acquiring  a system  under  the  following 

circumstances? 

*•  System  requires  12  years  to  plan,  design,  build, and  test. 

**  The  useful  life  of  majo,.  system  components  is  30  to  40  years. 

••  System  costs  $1  billion  to  build. 

••  The  cost  of  replacement  production  due  to  outages  is  $250,000 
to  $1  million  per  day. 

••  The  number  of  prime  contractors  and  major  suppliers  is  small. 

• • In-house  capability  is  oriented  to  operations  rather  than  to 
design,  development,  and  installation. 

• • Users  of  system  output  are  very  sensitive  to  interruptions  in 
output  and  price  of  output. 

1.5.3  Performance  Assurance  Programs  Selected  for  Review  and  Analysis 

Although  the  initial  goal  was  to  analyze  private-sector  as  well  as 
government  programs,  the  results  are  heavily  biased  by  the  character  and 
experience  of  government  programs.  Private -sector  concern  over  the  pro- 
prietary nature  of  its  programs  made  it  difficult,  in  the  short  time 
available  for  the  study,  to  gain  access  to  key  documents  defining 
performance  assurance  programs  of  private  organizations. 

Performance  assurance  programs  in  the  following  agencies  and  companies 
were  reviewed  and  analyzed: 

• U.S.  Department  of  Defense 

• National  Aeronautics  and  Space  Administration 

• Federal  Aviation  Administration  and  the  commercial  airline  industry 

• The  National  Transportation  Safety  Board 

• Urban  Mass  Transit  Administration 

• U.S.  Nuclear  Regulatory  Commission 

• The  Bell  System  (AT&T) 

Performance  assurance  programs  in  the  U.S.  Department  of  Energy,  such 
as  those  developed  for  application  to  the  Clinch  River  Breeder  Reactor  Pro- 
ject (CRBRP)  and  fossil  energy  projects,  were  reviewed  also;  but  they  are 
in  a preliminary  stage  and  yielded  no  results  considered  appropriate  for 
inclusion  in  this  report. 
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1.5.4  Key  Documents 


Key  documents  describing  mid  defining  formalized  performance  assurance 
programs  in  the  selected  agencies  are  identified  in  Table  1-2. 


1.6  GUIDE  TO  THIS  REPORT 

The  results  and  recommendations  are  presented  in  Chapter  Two.  A com- 
prehensive performance  assurance  program  to  be  considered  for  application  in 
the  electric  power  industry  is  outlined  there. 

The  information  gathered  during  the  interviews  and  from  pertinent 
references  is  presented  in  seven  appendixes.  Each  appendix  except  Appendix 
C describes  performance  assurance  programs  in  a single  agency  or  industry. 
Appendix  C deals  with  the  performance  assurance  programs  of  both  the 
Federal  Aviation  Administration  and  the  commercial  airline  industry.  Each 
appendix  is  organized  in  the  following  manner:  First,  the  historical 
development  and  general  scope  of  performance  assurance  practices  are 
described.  Second,  the  potential  applicability  of  specific  practices  to 
the  power  industry  is  addressed.  Third,  those  practices  which  seem  most 
worthy  of  further  consideration  are  described  in  more  detail.  Finally, 
some  indications  of  the  cost  and  effectiveness  of  the  program  from  which 
the  applicable  practices  were  drawn  are  offered. 


Table  1-2.  CITED  KEY  DOCUMENTS 


Agency 

Refe re nee 

Number 

Document 

DoD 

A-l 

DoD  Directive  5000.1  - Acquisition  of  Major 

Defense  Systems 

OMB 

A- 2 

OMB  Circular  A109  - Major  System  Acquisition 

DoD 

A- 3 

MIL  STD  785  - Reliability 

DoD 

A- 4 

MIL  STD  470  - Maintainability 

DoD 

A- 5 

DoD  Directive  5000. 3 - Test  and  Evaluation 

NASA 

B-l 

NHB  5300. 4 (IB)  (Formerly  NPC  200-2)  - Quality 
Program  Provisions  for  Aeronautical  and  Space 
System  Contractors 

NASA 

B-2 

NHB  5300.4  (1A)  (Formerly  NPC  250-1)  - Reliabil- 
ity Program  Provisions  for  Aeronautical  and 

Space  System  Contractors 

FAA 

C-l 

FAA-G-2100/lb  - Electronic  Equipment,  General 
Requirements 

FAA 

C-2 

Reliability /Maintainability  Systems  Engineering 
Program  Plan,  FAA,  Airways  Facilities  Service 

UMTA 

E-l 

MARTA  Reliability  Program  Plan  - Metropolitan 
Atlanta  Rapid  Transit  Authority 

NRC 

F-l 

Code  of  Federal  Regulations,  Title  10,  Part  50, 
Appendix  B,  Quality  Assurance  Criteria  for 

Nuclear  Power  Plants  and  Fuel  Processing  Plants 

F-2 

NUREG-75/087 , Section  17.1  - NRC  Standard  Review 
Plan,  Quality  Assurance  During  the  Design  and 
Construction  Phase 

F-3 

NUREG-75/087,  Section  17.2  - NRC  Standard  Review 
Plan,  Quality  Assurance  During  the  Operations 

Phase 

CHAPTER  TWO 


RESULTS  AND  RECOMMENDATIONS 


2.1  RESULTS  OF  THE  SURVEY 

The  characteristics  of  performance  assurance  in  the  selected  agencies, 
industries,  and  companies  are  summarized  in  Table  2-1.  Although  each  agency 
or  firm  among  those  surveyed  tends  to  emphasize  different  aspects  of  per- 
formance assurance  depending  upon  its  charter  and  history,  there  are  large 
areas  of  general  agreement.  The  overall  finding  is  that  most  performance 
assurance  programs  are  evolving  in  similar  ways  to  fulfill  a common  need. 
This  need  is  to  improve  the  effectiveness  of  the  contractual  arrangement 
between  operators  of  large,  interconnected,  and  expensive  systems  and  their 
system  acquisition  contractors  and  suppliers. 

2.1.1  The  Performance  Assurance  Function 

All  respondents  agree  that  performance  assurance  is  primarily  an 
acquisition  management  concept  aimed  at  identifying  and  correcting  poten- 
tial operating  deficiencies  early  in  the  acquisition  process.  Most  re- 
spondents point  out  that  system  acquisition  and  system  operation  are 
separate  and  distinct  management  functions  and  that  the  performance 
assurance  function  is  to  assure  that  operating  imperatives  are  addressed 
during  the  acquisition  process. 

2.1.2  Implementing  Performance  Assurance 

The  consensus  is  that  the  acquisition  contractual  document  is  the 
principal  vehicle  for  applying  a formal  performance  assurance  program;  i.e.  , 
most  respondents  emphasize  that  the  assurance  aspect  of  a formal  program  is 
contractual  in  nature.* 

Most  respondents  agree  that  a formal  approach  to  performance  assurance 
is  most  cost-effective,  particularly  in  situations  in  which  the  system 
acquisition  process  involves  many  subcontractual  arrangements,  involves 
large  and  continual  commitments  of  capital,  and  extends  over  many  years. 

In  such  circumstances,  the  majority  opinion  holds  that  it  is  important  to 


♦Respondents  in  regulatory  agencies  such  as  NRC  tend  to  emphasize  the  leg- 
islative aspect  of  assurance.  However,  even  regulatory  requirements  must 
ultimately  be  addressed  in  the  contractual  document. 
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Table  2-1.  CHARACTERISTICS  OF  PERFORMANCE  ASSURANCE  IN  SELECTED  AGENCIES,  INDUSTRIES,  AND  COMPANIES 
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Aircraft, 
engines,  and 
avionics 

10-20  years 

State  of  the 

art 

Daily  operation 
for  period  of 
10-16  hours 

Broad  use 

Limited 

No 

No 

Safety, 

reliability, 

maintainability, 

cost 

Certification 

authority 

No 

No 

No 

Yes 

Limited 

National 

Aeronautics 
and  Space 
Administration 

Missile, 
spacecraft, 
and  control 
systems 

A few  days  to 
several  years 

Pushes  state 

of  the  art 

Missions  vary 
from  a few 

hours  to 

continuous 
operat ion 

Very  limited 
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Reliability, 
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Combat  systems 

10-20  years 

Pushes  state 

of  the  art 

Missions  vary 
from  several 
hours  to 

continuous 

operation 

Limited 
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XXX 
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availability, 

cost 

Specifications 

applied 

through 
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Yes 

Yes 

Yes 

Yes 

Factor 
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System  Type 

Life  Charac- 
teristics 

Technology 

Status 

Operation 

Requirement 

Uses 

Standard 

Equipment 

Sponsors 
System  r&d 

Funds 

Acquisition 

Operates 

System 

Primary 

Concern 

Implementa- 

tion 

Authority 

Quantita- 
tive Goals 

Formal 

Program 

Formal 

Quantitative 
Demonstra- 
tion Test 

Qualifica- 
tion Testing 

Operational 

Feedback 
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codify  or  formalize  even  common-sense  practices  which  traditionally  have 
been  taken  for  granted,  and  to  define  responsibility  and  authority  for 
implementing  these  practices  in  an  explicit  way. 


2.1.3 


Delegation  of  Effort 


All  respondents  agree  that  it  is  important  to  divide,  apportion,  and 
share  the  performance  assurance  effort  among  all  participants  interested 
in  system  acquisition  and  operation.  Within  the  electric  power  industry, 
that  sharing  should  be  between  the  national  level,  the  acquiring  entities, 
and  the  contractors  and  suppliers. 


2. 1.3.1  National  Effort 


The  degree  of  public  vs.  private  control  over  system  performance 
varies  among  the  agencies  and  firms  surveyed  to  an  extent  that  precludes 
agreement  concerning  an  overall  national  effort.  However,  it  is  possible 
to  identify  elements  that  reflect  a consensus  on  policies  and  practices 
of  a majority.  A national  program  comprising  such  elements  is  described 
in  Table  2-2. 


2. 1.3. 2 Local  Effort 

All  agree  that  the  lead  authority  and  responsibility  for  implementing 
performance  assurance  should  rest  with  the  organization  responsible  for 
acquiring  the  system.  A minimal  effort  by  the  acquiring  entity  which  fits 
the  consensus  is  described  in  Table  2-3. 


2.1.4  The  Evolution  of  Performance  Assurance  Programs 

Although  performance  assurance  programs  have  evolved  differently  in 
all  agencies  and  firms  surveyed,  there  is  a perceptible  common  trend  in 
all. 


Typically,  the  first  step  is  an  attempt  to  utilize  failure,  mainte- 
nance, and  cost  experience  with  one  system  to  improve  the  design  of  an- 
other. Initially,  this  experience  is  transferred  in  an  informal  way  from 
operating  engineers  to  design  engineers  via  professional  societies,  trade 
journals,  or  person-to-person  contact. 

Subsequently, an  information  system  is  established  to  provide  quanti- 
tative but  generic  data  for  use  in  identifying  critical  problem  areas. 
These  data  are  then  used  by  design  engineers  to  estimate  potential 
performance. 

In  time,  the  information  system  and  data  base  are  refined  to  the 
point  where  statistically  valid  performance  prediction  becomes  possible. 
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Table  2-2.  RECOMMENDED  ELEMENTS  OF  A COMPREHENSIVE  NATIONAL* 
PERFORMANCE  ASSURANCE  EFFORT 


• Articulate  national  performance  goals  and  measures. 

• Estimate  the  value  and  cost  of  realizing  national  performance  goals. 

• Sponsor  the  development  of  training  programs. 

• Sponsor  the  development  of  guidelines  for  implementing  local-level 
programs . 

• Sponsor  the  development  and  refinement  of  performance  reporting 
systems. 

• Sponsor  the  development  and  refinement  of  specialized  techniques 
and  models. 

• Sponsor  the  development  of  centralized  test  facilities. 

• Sponsor  the  development  of  centralized  failure  analysis  facilities. 

• Develop,  in  cooperation  with  state  and  local  governments  and 

regional  cooperatives,  appropriate  incentives  for  improving 
performance. 

• Sponsor  product  improvement  research. 


*A  national  effort  does  not  necessarily  imply  a Federal  Government 
role.  For  example,  AT&T  centralizes  and  coordinates  the  Bell  System 
performance  assurance  program  through  Western  Electric  and  Bell 
Telephone  Laboratories.  A performance  assurance  program  for  appli- 
cation in  the  electric  power  industry  could  be  centralized  and 
coordinated  at  the  national  level  by  industry  associations. 


Table  2-3.  PERFORMANCE  ASSURANCE  EFFORT  BY  THE 
ACQUIRING  ENTITY* 


• Define  explicit  measures  of  cost-effective  operational  performance, 
considering  regulatory  requirements  where  necessary. 

• Introduce  quantitative  operational  performance  goals  into 
acquisition  contracts. 

• Require  acquisition  contractors  to  present  a plan  for  achieving 
performance  goals,  i.e.,  a performance  assurance  program  plan. 

• Review  and  evaluate  contractor's  and  supplier's  performance 
assurance  programs. 


*E.g.,  an  electric  power  utility. 


It  is  at  this  juncture  that  various  agencies  and  firms  decide  whether 
to  incorporate  quantitative  performance  goals  into  acquisition  contracts. 
Typically  they  decide  to  try,  and  they  seek  precedents  for  doing  so.  In 
almost  every  case,  the  precedents  can  be  ultimately  traced  to  DoD-sponsored 
activities. 

In  trying  to  incorporate  quantitative  performance  goals  into  the 
acquisition  process,  each  agency  and  firm  typically  faces  the  fundamental 
problem  of  demonstrating  conformance.  Each  has  five  interrelated 
alternatives : 

1.  Devise  tests  to  quantitatively  demonstrate  system  performance 
after  the  system  is  built. 

2.  Devise  tests  to  quantitatively  demonstrate  subsystem  performance 
as  the  system  is  designed. 

3.  Forget  the  need  to  demonstrate  performance  quantitatively;  test 
to  failure,  analyze,  and  fix. 

4.  Forget  the  need  to  demonstrate  performance  quantitatively;  rely 
on  engineering  judgment  and  review. 

5.  Rely  on  standard  equipment. 

Once  an  agency  or  firm  has  faced  these  choices,  it  becomes  possible 
to  develop  a mature  performance  assurance  program  involving  an  appropriate 
compromise  between  the  five  choices.  In  effecting  this  compromise,  it  is 
helpful  to  utilize  specialized  analytic  techniques  aimed  at  transferring 
and  translating  the  results  of  quantitative  measures  of  performance  at 
subsystem  levels  to  the  system  level  and  vice-versa.  These  analytic 
techniques  have  been  developed  and  are  being  further  developed  as  the 
five  choices  become  apparent  to  more  and  more  agencies  and  firms. 

At  this  juncture  in  the  evolution  of  a typical  performance  assurance 
program,  the  problem  of  monitoring  and  evaluating  contractual  performance 
becomes  paramount  due  to  the  necessity  of  evaluating  a mixture  of  quanti- 
tative and  qualitative  measures  throughout  the  acquisition  process.  Most 
respondents  solve  this  problem  by  periodically  evaluating  the  contractor's 
performance  assurance  program  against  the  contractor's  own  program  plan. 

In  a highly  evolved  program,  the  operating  agency  or  firm  requires 
a system  or  subsystem  performance  warranty.  The  warranty  approach  assigns 
a major  portion  of  the  responsibility  and  effort  for  assuring  system 
performance  to  contractors  and  subsystem  suppliers. 

The  ultimate  performance  assurance  program,  i.e.,  a very  highly 
evolved  one,  employs  sophisticated  models  for  trading  off  various  aspects 
of  system  performance.  A typical  evolution  to  the  highest  state  involves 
the  increasingly  cumulative  consideration  of: 

• Subsystem  reliability 

• Subsystem  maintainability 
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• System  availability 

• System  life-cycle  cost 

• Subsystem  redundancy 

• System  interconnection 

• System  redundancy 

2.1.5  Specifying  Performance  Goals 

Most  respondents  set  quantitative  performance  goals  in  their  requests 
for  proposals  as  a means  of  guiding  contractors  and  suppliers  in  developing 
an  appropriate  performance  assurance  program.  However,  all  recognize 
that  demonstration  of  performance  at  the  system  level  against  quantitative 
goals  is  very  expensive  and  often  impossible  from  a rigorous  mathematical 
viewpoint.  Therefore,  most  accept  a combination  of  analysis  and  demon- 
stration as  evidence  of  achievement. 

2. 1.5.1  Specifying  Reliability  Goals 

Reliability  goals  are  most  often  expressed  in  terms  of  mean  time 
between  failures  (MTBF)  or  failure  rate.  A typical  approach  is  to  specify 
the  mean  time  between  forced  outages  at  the  system  or  major  subsystem 
level,  leaving  it  to  the  acquisition  contractor  or  suppliers  to  allocate 
the  goal  to  lower  levels  in  the  system  and  present  a plan  for  demonstrating 
achievement . 

2. 1.5. 2 Specifying  Maintainability  Goals 

Maintainability  goals  are  typically  specified  in  terms  of  mean  time 
to  repair  (MTTR) . Quantitative  maintainability  goals  are  set  to  guide 
the  preparation  of  an  integrated  logistics  plan  or  a maintenance*  plan; 
i.e.,  quantitative  maintainability  goals  are  typically  set  to  stimulate 
the  identification  of  critical  maintenance  problems  early  in  the  system 
acquisition  process.  Most  respondents  do  not  require  demonstration  of 
achievement,  but  evaluate  the  methods  and  techniques  for  minimizing  down- 
time. There  is  growing  interest  in  methods  and  devices  for  detecting 
signs  of  abonormal  wear  and  adjusting  maintenance  actions  to  these  signs. 

2.1. 5. 3 Specifying  Availability  Goals 

Most  respondents  do  not  set  quantitative  availability **  goals  although 
all  point  out  the  need  to  trade  off  reliability  and  maintainability  goals 
against  other  performance  measures  such  as  life-cycle  cost.  There  is  a 
growing  tendency, as  agencies  and  firms  become  more  aware  of  differences  in 
acquisition  and  operating  requirements,  to  specify  availability  or  some  other 


♦Maintenance,  as  used  herein,  includes  preventive  maintenance,  repair, 
and  replacement.  For  applications  in  the  electric  power  industry, 
maintenance  would  include  all  actions  related  to  the  overall  goal  of 
minimizing  the  duration  of  forced  and  scheduled  outages. 

**One  simple  measure  of  availability  is  MTBF/ (MTBF  + MTTR) . 
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combined  measure  of  reliability  and  maintainability  and  then  determine 
reliability  and  maintainability  goals  as  necessary  to  minimize  life-cycle 
costs. 

2. 1.5.4  Specifying  Life-Cycle-Cost  Goals 

Although  life-cycle-cost  goals  are  rarely  specified  in  acquisition 
contracts,  all  respondents  agree  that  there  is  a trend  in  that  direction. 
Life-cycle  costs  include  all  acquisition  and  operating  costs. 

2.1.6  Specifying  Redundancy 

Reliability,  maintainability,  and  availability  may  be  improved  through 
product  improvement  or  by  providing  redundant  alternatives.  Redundancy 
may  be  provided  at  any  level  in  the  system  hierarchy.  Although  redundancy 
is  typically  specified  only  for  safety-critical  items,  all  respondents 
recognize  the  importance  of  being  able  to  evaluate  product  improvement  vs. 
redundancy  alternatives  as  they  affect  life-cycle  cost. 

2.1.7  Specifying  Operating  Profiles,  Levels  of  Stress,  and 
Derating  Requirements 

All  respondents  emphasize  the  importance  of  specifying  operation 
profiles,  levels  of  stress,  and  derating  requirements  in  the  acquisition 
contract.  Most  specify  operating  environmental  conditions  precisely  and 
in  quantitative  terms,  if  possible. 

2.1.8  Specifying  Rates  of  Technological  Change 

Few  respondents  specify  the  degree  and  rate  of  technological  change 
except  indirectly  by  calling  out  standard  equipments. 

2.1.9  Requiring  a Performance  Assurance  Program  Plan 

All  respondents  require  contractors  and  suppliers  to  present  formal 
performance  assurance  plans  as  part  of  their  proposals.  Typically,  the 
acquiring  entity  prescribes  program  requirements  in  the  request  for 
proposal  by  calling  out  appropriate  performance  assurance  program  standards 
or  guides. 

Among  the  items  which  all  respondents  agree  should  be  addressed  in 
the  contractual  program  plan  are  those  shown  in  Table  2-4. 

2.1.10  Cost  and  Effectiveness  of  Contractual  Performance  Assurance 

It  was  not  possible  to  obtain  detailed  information  concerning  the  cost 
and  effectiveness  of  applying  contractual  performance  assurance  in  the  short 
time  available  for  the  study.  The  number  of  factors  to  be  considered  is  too 
large.  Nevertheless,  it  is  possible  to  draw  some  general  and  qualitative 
conclusions. 


Performance  Assurance  Management  and  Evaluation  Plan 

Training  and  Indoctrination  Plan 

Subcontractor  and  Supplier  Control  Plan 

Design  Specifications 

• • Operating  specifications 

••  Environmental  specifications 

••  Reliability  specifications 

••  Maintainability  specifications 

••  Availability  specifications* 

••  Life-cycle-cost  specifications* 

••  Redundancy  specifications* 

Performance  Prediction  and  Estimation  Plan 
• • Requirements 
• • Techniques 
• • Data 

Iterative  Failure  Modes  and  Effects  Analysis  (FMEA)  Plan 
Design  Review  Plan 

Problem  Failure  Reporting  and  Correction  Plan 
Standardization  of  Design  Practices 
Parts  and  Materials  Program  Plan 
Configuration  Management  Plan 
Test  Plan 

• • Quantitative  demonstration  tests 
••  Test  to  failure,  analyze,  and  fix 
••  Qualification  tests 

• • Performance  assurance  documentation  plan 

••  Warranty  plan*  or  plan  for  contracter  and  supplier  involve 
ment  during  operation. 


2.1.10.1  Cost  of  Contractual  Performance  Assurance 


The  incremental  cost  of  applying  a formal  performance  assurance  program 
during  the  acquisition  of  Typical  DoD  and  NASA  systems  adds  from  one  to  ten 
percent  to  the  cost  of  the  system.  However,  there  are  indications  that  the 
incremental  cost  of  applying  performance  assurance  during  the  acquisition 
of  mass  transit  systems  adds  only  about  0.01  to  0.2  percent  to  the  cost. 
Apparently,  mass  transit  systems,  like  electric  power  plants,  contain  many 
standard  components  and  a large  proportion  of  relatively  massive  and  immobile 
structures  that  add  considerably  to  the  system  cost  wihtout  adding  much  to 
the  performance  assurance  burden. 

2.1.10.2  Incremental  Performance  Assurance  Cost-Effectiveness 

Most  respondents  agree  that  the  cost-effectiveness  is  greatest  if  con- 
tractual performance  assurance  is  applied  as  early  as  possible  in  the 
acquisition  process.  Indications  are  that  pre-contract  planning  and 
iterative  design  technique*  offer  the  greatest  payoff  for  the  smallest 
expenditure. 

All  respondents  recognize  that  the  cost  of  improving  system  performance 
increases  rapidly  with  system  MTBF  and  the  reciprocal  of  system  MTTR.  Thus 
the  cost-effectives  decreases  rapidly  as  the  system's  performance 
improves. ** 

Most  respondents  would  agree  that  the  cost-effectiveness  or  performance 
assurance  increases  as  system  acquisition  cost  increases.  Our  rationale 
for  this  rule  is  simply  that  outages  are  more  costly  as  the  payback  burden 
increases.  A corresponding  rule,  derivable  from  the  above,  is  that  cost- 
effectiveness  of  performance  assurance  decreases  as  the  payback  schedule 
is  extended. 


2 . 2  RECOMMENDATIONS 

Before  deciding  whether  or  not  a comprehensive  performance  assurance 
program  based  on  the  results  of  the  survey  is  applicable  in  the  electric 
power  industry,  it  would  be  helpful  to  address  the  following  questions: 

1.  To  what  extent  do  electric  utilities  employ  performance  assurance 
practices  similar  to  those  summarized? 


♦Iterative  design  technique  involves  continual  or  stage-by-stage  performance 
analysis  and  testing  during  the  design  process. 

♦♦There  are  possible  exceptions  to  this  rule.  If  the  dominate  measure  of 
effectiveness  involves  loss  of  life  or  social  risk,  it  is  possible  to 
argue  that  almost  any  performance  assurance  cost  is  justifiable. 
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2.  What  are  the  major  economic,  institutional,  and  technical  con- 
straints in  the  electric  power  industry  that  might  preclude  or 
impede  the  application  of  performance  assurance  practices  such 
as  those  summarized? 

3.  Assuming  that  the  answers  to  the  above  questions  indicate  that 
a comprehensive  program  such  as  that  summarized  can  be  suitably 
modified  for  application  in  and  acceptance  by  the  electric  power 
industry,  how,  at  what  rate,  and  by  whom  should  the  program  be 
implemented? 

Because  the  system  acquisition  contract  is  such  an  important  mechanism 
in  implementing  performance  assurance,  it  is  recommended  that  the  U.S. 
Department  of  Energy  sponsor  the  development  of  model  performance  assurance 
contracts  for  voluntary  use  by  the  electric  power  industry.  The  development 
effort  should  include: 

• Analysis  of  current  contractual  documents  used  by  electric  power 
utilities  in  acquiring  new  baseload  capacity 

• Comparison  of  contractual  performance  assurance  practicies  in  the 
electric  power  industry  to  those  found  effective  by  others 

• Estimation  of  the  costs  and  benefits  of  implementing  selected 
contractual  practices  in  the  electric  power  industry 

• Formulation  of  model  contractual  documents  for  review  by  electric 
utlities  and  their  acquisition  contractors 

• Definition  of  strategies  for  stimulating  the  expanded  use  of  con- 
tractual performance  assurance  requirements  by  electric  utilities 
and  their  acquisition  contractors 
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APPENDIX  A 


PERFORMANCE  ASSURANCE  AS  PRACTICED  BY 
THE  U.S.  DEPARTMENT  OF  DEFENSE 


1.  HISTORICAL  DEVELOPMENT 

The  current  DoD  performance  assurance  activities  had  their  origin  in 
the  early  1930s.  Major  General  Leslie  E.  Simon*  traces  one  line  of  develop- 
ment from  the  publication  of  Dr.  Walter  A.  Shewhart's  "Economic  Control  of 
Manufactured  Product"  in  1931**  recommending  the  application  of  statistical 
methods  to  controlling  quality  (SQC) , an  approach  which  was  tried  at  Pica- 
tinny  Arsenal  in  1936  and  applied  throughout  World  War  II.  Near  the  end 
of  that  war,  statistical  and  probabilistic  methods  were  applied  successfully 
by  operations  researchers  (OR)  assigned  to  the  staffs  of  military  commanders 
to  analyze  alternative  operational  plans.  By  the  end  of  the  war,  the  SQC 
and  OR  concepts  became  joined  in  what  subsequently  became  the  basis  for 
reliability  assurance.  By  1948,  quantitative  methods  were  being  applied  by 
the  Rand  Corporation  to  evaluate  the  reliability  of  guided  missiles. t By 
1950,  Dr.  Robert  Lusser  had  enunciated  a concept  of  reliability  for  applica- 
tion in  achieving  reliability  in  guided  missiles. ft  Subsequently  the  prob- 
lem of  organizing  to  manage  reliability  was  addressed#  and,  by  1952,  reli- 
ability assurance  was  emerging  as  a quantitative  discipline  separate  and 
distinct  from  the  quality  control  discipline. 


*L.  E.  Simon,  "The  Relation  of  Engineering  to  Very  High  Reliability", 
Proceedings , Tenth  National  Symposium  on  Reliability  and  Quality  Control, 
1964,  pp  226. 

**W.  A.  Shewhart,  "Economic  Control  of  Quality  of  Manufactured  Product",  D. 
Van  Nostrand  Co.,  N.Y.,  1931. 

tD.  J.  Davis,  "Concepts  and  Methods  for  Investigation  of  Guided  Missile 
Reliability",  Rand  Corp. , Report  R-107,  November  1948. 
t+Naval  Air  Missile  Test  Center,  "A  Study  of  Methods  for  Achieving  Reli- 
ability of  Guided  Missiles",  Tech.  Report  No.  75,  July  1950. 

#"Final  Report  of  Subcommittee  on  Reliability  of  the  RDB  Guided  Missile 
Committee",  Department  of  Defense,  March  1952. 
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During  World  War  II  complex  electronic  systems  were  created  which  in 
many  cases  were  quite  unreliable.  This  problem  led  to  a number  of  studies 
in  the  late  1940s  and  early  1950s  which  investigated  the  failure  patterns 
of  these  systems.  These  studies  culminated  when  the  Advisory  Group  on 
Reliability  of  Electronic  Equipment  (AGREE) , Office  of  the  Assistant  Sec- 
retary of  Defense,  in  1957  issued  a report  on  recommendations  for  quanti- 
tative control  of  reliability  including  testing.  From  these  efforts,  a 
series  of  specifications  were  issued  by  the  various  military  services  as 
part  of  system  acquisition  establishing  reliability  as  a controlled 
parameter. 

The  reliability  development  was  followed  in  the  early  1960s  by  the 
inclusion  of  maintainability  as  a contractually  specified  parameter.  In  a 
series  of  studies  the  military  investigated  the  nature  of  equipment  mainte- 
nance; those  studies  led  to  the  formation  of  specifications  and  supporting 
techniques. 

In  the  1963-65  period,  the  Air  Force  sponsored  the  "Weapon  System 
Effectiveness  Industry  Advisory  Committee  (WSEIAC) , whose  purpose  was  to 
integrate  reliability,  maintainability,  and  performance  characteristics  into 
an  overall  concept  called  system  effectiveness.  The  developed  concept  met 
the  objectives  stated  but  due  to  the  complexity  of  applications,  limited  use 
has  been  made  to  date.  No  specifications  directly  resulted  from  this  effort 

As  progress  was  made  in  reliability  and  maintainability  as  they  relate 
to  mission  effectiveness,  it  soon  became  apparent  that  they  also  seriously 
affected  life-cycle  cost.  In  recognition  of  this  concern  for  life-cycle 
cost  and  mission  effectiveness  along  with  the  need  for  thorough  testing,  an 
approach  to  meet  these  needs  was  given  in  DoD  Directive  5000.1,  Acquisition 
of  Major  Defense  Systems  issued  in  1971  (Key  Document  A-l) . This  document 
called  for  a series  of  major  program  review  points  at  which  the  performance, 
cost,  and  program  risks  were  assessed  before  authorization  to  proceed  to  the 
next  phase  was  given.  Performance  assurance  considerations  form  a vital 
part  of  the  review  process  due  to  its  impact  on  both  effectiveness  and 
life-cycle  cost. 

In  1976  the  Office  of  Federal  Procurement  Policy  issued  OMB  Circular 
A109  (Key  Document  A-2) , which  expanded  the  DoD  phased  concept  by  requiring 
the  detailed  study  of  mission  needs  prior  to  embarking  on  system  develop- 
ment. In  essence  the  mission  need  should  generate  system  requirements 
rather  than  the  system  defining  the  mission.  Zero  based  budgeting,  which 
has  emerged  in  this  same  period,  requires  the  investigation  of  alternate 
concepts  and  their  attendant  benefits  and  costs.  Such  analysis  focuses 
much  attention  on  performance  and  cost  and  their  underlying  drivers. 
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SCOPE  OF  CURRENT  DOD  PRACTICES 


2 . 1 Introduction 

The  DoD  performance  assurance  program  consists  of  two  major  areas. 

First,  there  is  the  activity  associated  with  system  research  and  development 
and  acquisition  or  procurement.  The  second  area  is  associated  with  system 
operation  and  maintenance.  The  practices  in  each  area  will  be  reviewed  in 
Sections  2.2  and  2.3. 

The  DoD  assurance  program  is  continuing  to  change  and  it  is  difficult 
to  adequately  characterize  its  nature.  There  also  exist  differences  among 
the  individual  services  in  emphasis  and  detailed  approach.  The  current 
program  was  developed  during  an  era  characterized  by  very  poor  reliability 
levels.  This  was  also  an  era  during  which  the  principal  emphasis  was  on 
developing  systems  with  high  technical  performance  characteristics.  More 
recently,  cost  has  become  an  increasingly  important  consideration.  The 
current  trend  is  to  place  at  least  as  much  emphasis  on  reducing  system 
life-cycle  costs  as  on  improving  technical  performance. 

As  background  to  the  concepts  used,  it  is  of  interest  to  review  the 
organization  structure  of  the  services  as  it  relates  to  the  system  life  cycle 
and  to  the  type  systems  which  they  acquire. 

Within  the  military  services,  the  system  acquisition,  maintenance  and 
support,  and  the  operations  functions  are  separate  and  distinct  entities. 

The  operations  group  generates  requirements  for  a system  that  are  addressed 
during  design,  development,  and  procurement.  Upon  installation  and  delivery 
the  operating  activity  uses  the  system  and  provides  on-site  maintenance. 

Their  operation  is  supported  by  a logistics  activity  which  provides  spare 
parts  and  overhaul  (off-site)  maintenance.  Although  this  structure  permits 
specialists  to  operate  in  each  area,  communication  of  requirements  between 
organizations  is  a continuing  problem. 

It  is  also  important  to  note  that  many  of  the  systems  employed  within 
the  military  were  designed  and  developed  expressly  for  those  services.  They 
accordingly  sponsor  the  underlying  research  and  development  leading  to  the 
production  item.  Efforts  are  being  made  to  use  commercial  off-the-shelf 
items  but  it  is  expected  that  most  of  the  combat-rciated  equipment  will 
continue  to  be  specially  designed. 


2.2  Acquisition  Procedures 


The  Department  of  Defense  tailors  its  performance  assurance  program 
to  meet  specialized  requirements  by  calling  out  specifications  and  standards 
in  the  acquisition  contract.  An  example  of  a typical  call-out  is  shown  in 
Figure  A-l. 


APPLICABLE  DOCUMENTS 


All  documents  referenced  under  this  paragraph  shall  be  applicable  only 
as  specified  within  this  Statement  of  Work.  In  the  event  of  any  conflict 
in  requirements  between  applicable  documents,  the  following  order  of  prece- 
dence shall  prevail:  (a)  This  Statement  of  Work;  (b)  ARINC  documents; 

(c)  FAA  documents;  (d)  military  documents;  and  (e)  other  documents.  Docu- 
ments listed  herein  shall  be  the  latest  revision  as  of  the  date  of  contract 
solicitation. 


SPECIFICATIONS 

MIL-D-1000 

Drawing,  Engineering  and  Associated  List 

MIL-S-5002 

Surface  Treatments  and  Inorganic  Coatings  for 
Metal  Surfaces  of  Weapons  Systems 

MIL-W-5086 

Wire,  Electric,  PVC  Insulated  Copper  or  Copper 
Alloy 

MIL-B-5087 

Bonding,  Electrical,  and  Lighting  Protection 
for  Aerospace  Systems 

MIL-W-5088 

Wiring,  Aircraft,  Selection  and  Installation  of 

MIL-E-5400 

Electronic  Equipment,  General  Specification  for 

MIL-C-5414 

Computer,  Air  Navigation,  Dead  Reckoning,  Type 
MB-4A  and  CPU-26A/P  (C-141  only) 

MIL-E-6051 

Electromagnetic  Compatibility  Requirements, 
Systems 

MIL-E-7016 

Electrical  Load  and  Power  Source  Capacity, 
Aircraft  Analysis  of 

MIL-E-7080 

Electrical  Equipment,  Aircraft,  Selection  and 
and  Installation  of 

MIL-F-7179 

Finishes  and  Coatings,  Protection  of  Aerospace 
Weapon  Systems,  Structures  and  Parts,  General 
Specification  for 

MIL-M-7793 

Meter,  Time  Totalizing 

MIL-I-8700 

Installation  and  Test  of  Electronic  Equipment 
in  Aircraft,  General  Specification  for 

MIL-A-8806 

Sound  Pressure  Levels  in  Aircraft,  General 
Specification  for 

MIL- A- 8860 

Modification  for  Aircraft  Strength  and  Ridigity, 
General  Specification  for 

(continued) 


Figure  A-l.  TYPICAL  SPECIFICATION  AND  STANDARD  CALL-OUT  FOR  DOD 
EQUIPMENT  PROCUREMENT 


SPECIFICATIONS 

(continued) 

MIL-A-8865 

Aircraft  Strength  and  Rigidity,  Miscellaneous 
Loads 

MIL-A-8868 

Aircraft  Strength  and  Rigidity,  Data  and 

Reports 

MIL-Q-9858 

Quality  Program  Requirements 

MIL- D-9898 

Drawing,  Tube  Bend 

MIL-N-18307 

Nomenclature  and  Identification  for  Electronic, 
Aeronautical  and  Aeronautical  Support  Equipment 
Including  Ground  Support  Equipment 

MIL-W-25140 

Weight  and  Balance  Control  System  for  Airplanes 
and  Rotocraft 

MIL-I-25992 

Indicator,  Bearing  Distance  and  Heading  ID- 
526A/ARN  and  ID-798/ARN  (C-141  only) 

MIL-C-27500 

Cable,  Electrical,  Shielded  and  Unshielded 

MIL- 1-27848 

Indicator,  Horizontal  Situation,  AQU-4/A 
(C-141  only) 

MIL-C- 38037 

Computer,  Central  Air  Data,  CPU-43/A  (C-141 
only) 

MIL-C-38999 

Connector,  Electrical,  Circular,  Miniature, 

High  Density  Quick  Discount,  Environmental 
Resistant,  Removal  Crimp  Contacts 

MIL-M-43719 

Marking  Materials  and  Markers,  Adhesive, 
Elastomeric , Pigmented 

MIL-I-45208 

Inspection  System  Requirements 

MIL-H-46855 

Human  Engineering  Requirements  for  Military 
Systems,  Equipment  and  Facilities 

MIL-C-81659 

Connector,  Electrical,  DPX 

MIL-W-81044 

Wire,  Electric,  Copper  or  Copper  Alloy 

MIL-C-83723 

Connector,  Electrical,  Circular,  Environmental 
Resisting,  General  Specification  for 

MIL-C-5191 (V) 

Pitot  Static  True  Air  Speed  Computer  and 
Transmitter  (C-135) 

MIL-G-25591B 

Gyroscope,  Rate,  Switching,  Type  MC-1 

MIL-C-8780A 

N-l  Compass  System  (C-135) 

MIL-C-8412A 

J-4  Compass  System  (C-135) 

Figure  A-l.  (continued) 
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(continued) 


SPECIFICATIONS 

(continued) 

MIL-C-38240 

Altitude  Computer,  Altitude  Encoder,  CPU-66/ 

A-l,  Bendix  (C-135) 

MIL-G-25597 

MD-1  Vertical  Gyro  (C-135) 

STANDARDS 

MIL-STD-100 

Engineering  Drawing  Practices 

MIL-STD-130 

Identification  Markings  of  U.S.  Military 

Property 

MIL- STD-14 3 

Specifications  and  Standards,  Order  of 

Precedence  for  the  Selection  of 

MIL-STD-454 

Standard  General  Requirements  for  Electronic 
Equipment 

MIL- STD-704 

Electrical  Power  Aircraft,  Characteristics  and 
Utilization  of 

MIL- STD- 721 

Definition  of  Effectiveness  - Terms  for 
Reliability,  Maintainability,  Human  Factors 
and  Safety 

MIL-STD-756 

Reliability  Prediction 

MIL-STD-781 

Reliability  Tests,  Exponential  Distribution 

MIL-STD-785 

Reliability  Program  for  Systems  and  Equipment 
Development  and  Production 

MIL-STD-863 

Preparation  of  Wiring  Data 

MIL-STD-882 

Systems  Safety  Program  for  Systems  and  Asso- 
ciated Subsystems  and  Equipment,  Requirements 
for 

MIL-STD-1472 

Human  Engineering  Design  Criteria  for  Military 
Systems,  Equipment,  and  Facilities 

MIL- STD- 1521 

Technical  Review  and  Audits  for  Systems, 
Equipment  and  Computer  Programs 

MIL-STD-749 

Military  Standard  Preparation  and  Submission 
of  Data  for  Approval  of  Nonstandard  Parts 

(continued) 


Figure  A-l . (continued) 


HANDBOOKS 

MIL-HDBK-217 

Strength  of  Metal  Aircraft  Elements 

MIL-HDBK-217 

Reliability  Stress  and  Failure  Rate  Data  for 
Electronic  Equipment 

AFSC  DH  Series 

1-4,  1-6,  2-1, 
and  2-2 

Design  Handbooks  for  Aerospace  Systems 

MANUALS 

TACM/PACAF/ 

USAFEM  55-40 

Computer  Air  Release  Systems  Procedure 

TD-3  (Formerly 
AFSCM/AFLCM  310-1) 

DOD  Authorized  Data  List 

AFM  800-XX 

Computer  Resources  Acquisition  and  Support 

AIR  FORCE  REGULATIONS 

AFR  80-28 

Engineering  Inspections 

AFR  800-14 

Management  of  Computer  Resources  in  Systems 

Figure  A-l.  (continued) 
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2.2.1  Reliability  Programs 


TWo  key  DoD  documents  controlling  performance  assurance  are  MIL-STD-785, 
Reliability,  and  MIL-STD-470,  Maintainability  (Key  Documents  A-3  and  A-4 ) . 

The  approaches  used  in  both  areas  are  similar;  but  since  reliability  has  re- 
ceived greater  attention,  we  will  focus  on  this  program. 

The  major  elements  for  reliability  control  are  summarized  from  MIL-STD- 
785  in  the  following  subsections. 

2. 2. 1.1  Reliability  Program.  The  contractor  is  required  to  estab- 
lish and  maintain  an  effective  reliability  program  that  is  planned,  inte- 
grated, and  developed  in  conjunction  with  other  design,  development,  and 
production  functions  to  permit  the  most  economical  achievement  of  overall 
program  objectives.  The  required  reliability  program  involves  consideration 
of  management  and  technical  resources,  plans,  procedures,  schedule,  and 
controls  for  the  work  needed  to  assure  achievement  of  reliability  require- 
ments. The  program  is  adjusted  to  suit  the  type  and  phase  (design,  develop- 
ment, or  production)  of  the  procurement.  DoD  requires  that  the  program  be 
consistent  with  the  severity  of  the  mission  requirements,  the  complexity 

of  the  design,  the  need  for  commonality,  the  quantity  under  procurement, 
and  manufacturing  imperatives.  The  required  program  must  assure  reliability 
involvement  throughout  all  aspects  of  the  design,  development,  and  production 
as  necessary  to  meet  the  contractual  reliability  requirements. 

The  contractor's  proposed  program  plan  describes  how  he  plans  to  conduct 
the  reliability  program  to  meet  the  requirements  of  the  request  for  proposal 
and  the  statement  of  work,  in  order  to  comply  with  applicable  reliability 
program  elements.  The  plan  is  submitted  as  a separate  and  complete  entity 
within  the  contractor's  proposal.  The  reliability  program  plan,  as  approved 
by  the  procuring  activity,  is  incorporated  into  the  contract  and  becomes 
the  basis  for  contractual  compliance. 

2. 2. 1.2  Quantitative  Requirements.  The  system's  mission-responsive 
reliability  requirements  and  objectives  are  specified  contractually.  In 
addition,  the  minimum  acceptable  reliability  requirements  for  the  hardware 
are  specified.  Quantitative  hardware  reliability  requirements  for  all  major 
subsystems  and  equipments  are  included  in  appropriate  sections  of  the  system 
and  end-item  specifications.  DoD  requires  that  the  values  not  established 

by  the  procuring  agency  be  established  by  the  system  or  equipment  contractor 
at  a contractually  specified  control  point  prior  to  detail  design. 

2. 2. 1.3  Reliability  Demonstration.  DoD  requires  that  the  achieve- 
ment of  minimum  acceptable  hardware  reliability  requirements  be  demonstrated 
by  means  of  tests  and  analyses  specified  in  the  contract. 


2. 2. 1.4  Standards  Program.  The  DoD  has  attempted  to  use  standard 
items  (equipment,  modulas,  components,  etc.)  as  a means  of  achieving  system 
reliability.  The  concept  is  that  by  the  use  of  a standard  item  the  infant 
mortality  experienced  for  each  new  type  of  item  is  eliminated.  There  are 
also  cost  advantages  that  conceptually  make  this  approach  very  appealing. 

Major  progress  has  been  made  by  the  services  in  developing  and  using  stan- 
dard items  at  the  part  or  component  (e.g. , resistor,  integrated  circuit)  and 
at  the  equipment  (e.g.,  TACAN,  IFF)  levels.  Although  some  efforts  have  been 
directed  at  forming  and  using  standard  electronic  modules,  the  utility  of 
this  approach  has  not  been  proven. 

2. 2. 1.5  Test  Programs.  The  basic  thrust  of  the  current  DoD  reli- 
ability and  maintainability  programs  as  they  are  now  evolving  is  the  per- 
formance of  a series  of  design  tests  to  determine  the  ability  of  the  item 

to  operate  for  some  time  in  the  face  of  mission  environments.  Although  the 
contractors  who  normally  supply  the  DoD  have  available  to  them  the  necessary 
test  facilities,  the  DoD  has  built  and  maintains  special  test  facilities 
used  as  part  of  the  system  development  process. 

As  part  of  the  DoD  current  "fly  before  buy"  concept  (DoD  5000.1) , 
attention  has  been  given  to  testing  prototype  systems  prior  to  proceeding 
with  production.  To  assist  in  the  evaluation  of  prototype  as  well  as  initial 
operational  assessment  of  first  production  items,  the  DoD  has  established  test 
agencies  within  the  military  services  whose  purpose  is  to  provide  an  inde- 
pendent assessment  of  the  new  system. 


2 . 3 Operational  Concepts 

The  performance  assurance  program,  as  it  applies  to  operations,  is 
defined  primarily  by  the  maintenance  management  procedures  in  a series  of 
directives  and  manuals.  In  essence,  the  system  provides  for  reporting  the 
occurrence  of  maintenance  activity  which  leads  to  the  identification  by 
system  components  of  cost,  man-hours,  parts  cost,  and  occurrence  rates. 

An  exception  reporting  system  identifies  the  top  contributors  to  unsatis- 
factory performance  for  each  major  system. 

The  cognizant  system  manager  reviews  these  reports  and  if  he  deems 
items  appearing  in  the  reports  sufficiently  bad,  he  may  institute  corrective 
actions . 

The  maintenance  data  collected  are  also  available  as  an  aid  to  estab- 
lishing requirements  for  new  similar  systems.  With  additional  data  process- 
ing, the  services  also  use  these  data  to  revise  and  update  their  prediction 
handbook  (MIL-HDBK-217)  and  data  banks,  such  as  RAC  and  FARADA. 
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3.  APPLICABILITY  OF  DOD  METHODS  TO  THE  ELECTRIC  POWER  INDUSTRY 
3.1  Acquisition  Concepts 

3.1.1  Reliability-Maintainability  Program 

The  DoD  requires  that  an  organized  reliability  and  maintainability 
effort  be  defined.  Program  elements  have  included  activities  such  as: 

• Management  and  control 

• Subcontractor  and  supplier  program 

• Program  review 

• Design  analysis 

• Reliability  analysis 

• Parts  reliability 

• Failure  modes  and  effects  analysis 

• Critical  item  control 

• Storage  and  handling  impact  analysis 

• Design  review 

The  DoD  is  reemphasizing  the  design- related  activities  within  the 
program.  The  concept  of  test-analyze-fix  during  the  design-development 
phase  is  being  stressed  as  an  approach  which  produces  true  reliability 
and  maintainability  growth. 

The  electric  power  industry  could  benefit  from  an  organized  program 
of  design-related  activities  directed  toward  assessing  the  capability  of 
systems  to  operate  satisfactorily  in  their  intended  environments  for  de- 
sired periods  of  time  imposed  by  prescribed  maintenance  expenditure.  Each 
system  acquisition  is  different  and  presents  differing  risks  and  uncer- 
tainties; therefore,  it  is  necessary  to  define  a general  approach  which 
may  be  tailored  to  fit  each  specific  situation. 

It  is  important  that  the  system  reliability  and  maintainability  oper- 
ational requirements  be  quantitatively  identified,  allocated  to  subsystems, 
and  tracked  throughout  development,  production,  and  operation.  The  inclu- 
sion of  performance  assurance  parameters  as  contractual  requirements  is 
desirable,  but  the  requirement  to  demonstrate  compliance  must  be  practical. 

Reliability  demonstration  using  statistical  plans*  such  as  those  called 
for  in  MIL-STD-781  for  system  acquisition  can  be  very  costly,  especially 
for  low  production  items  with  moderate  MTBF  (100  hours) . The  DoD  has 


*A  statistical  test  plan  involves  the  use  of  mathematical  statistics  to 
calculate  how  long  the  system  must  function  without  failure  in  order  to 
meet  a quantitative  goal  (e.g.,  MTBF)  at  a specified  level  of  confidence. 
Typically,  the  system  must  operate  without  failure  for  a period  sub- 
stantially longer  then  the  MTBF  goal  in  order  to  pass  the  test. 
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substituted,  in  some  cases,  a limited  failure-free  operation  test  (burn-in) 
in  lieu  of  the  formal  test.  Also,  it  has  used  warranty  as  an  alternative  to 
demonstration  test,  relying  on  incentive  (positive  and  negative)  for  reli- 
ability achievement.  It  would  appear  that  the  power  industry  could  also 
make  use  of  a similar  policy. 

Maintainability  demonstration  is  not  nearly  as  costly  as  reliability. 
However,  in  many  situations  it  is  viewed  as  a less  critical  system  parameter 
and  the  developing  agency  may  choose  not  to  employ  demonstration.  It  is 
recommended  that  the  electric  power  industry  review  the  value  of  maintain- 
ability verification  and  determine  its  applicability  on  a case-by-case  basis. 

3.1.2  Standards 


Although  conclusive  evidence  is  not  readily  available,  the  DoD  appears 
to  have  experienced  its  greatest  success  with  standardization  at  the  equip- 
ment and  part-component  levels.  Standardization  above  these  levels  seems 
to  become  technically  obsolete  rapidly  and  is  complex  to  administer.  The 
electric  power  industry  could  benefit  from  the  development  of  recommended 
standards  if  the  appropriate  levels  can  be  defined. 

3.1.3  Test  Programs 

Tests  in  simulated  operating  environments  form  a vital  part  of  the  DoD 
approach  which  is  receiving  greater  emphasis.  To  meet  these  test  require- 
ments, the  DoD  has  developed  specialized  facilities.  The  possibility  of 
developing  similar  facilities  and  an  appropriate  organizational  structure 
should  be  studied  for  application  in  the  electric  power  industry. 

3 . 2 Operational  Assessment 

The  DoD  has  structured  maintenance  management  data  systems,  within  the 
several  services,  to  help  collect  and  analyze  field  performance  data. 

The  development  of  a comparable  data-gathering  and  feedback  system 
should  form  a vital  part  of  the  electric  power  performance  assurance  program. 
The  aim  should  be  to  enhance  existing  data  bases  and  establish  an  on-going 
operational  assessment  program  which  can  identify  problems  with  existing 
systems  and  provide  data  for  defining  future  requirements. 

4.  DESCRIPTION  OF  APPLICABLE  PRACTICES 

Highlights  of  the  applicable  acquisition  practices  and  operational 
assessment  methods  are  reviewed  in  this  section.  Also,  some  of  the  changing 
trends  emerging  within  the  DoD  approach  to  performance  assurance  are  noted. 
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4.1  Current  Acquisition  Concepts 

Currently,  DoD  treats  reliability  and  maintainability  as  quantitative 
parameters  specified  as  part  of  the  development  and  production  contract. 

A series  of  handbooks  and  standards  have  been  developed  to  provide  the 
mechanism  for  this  process.  Table  A-l  describes  the  essential  elements  of 
these  major  DoD  documents. 

4.1.1  Test  and  Demonstration 

It  should  be  noted  that  the  documents  put  major  emphasis  on  requiring 
the  contractor  to  perform  special  tasks  as  part  of  his  development  effort 
and  then  to  demonstrate  that  his  product  has  met  any  established  contractual 
reliability  and  maintainability  requirements  by  performing  a demonstration 
test.  Such  tests  may  be  applied  as  part  of  engineering  development  as  well 
as  continuing  production  demonstration.  Due  to  the  high  cost  of  demonstra- 
tion programs,  a number  of  recent  DoD  programs  have  made  use  of  burn-in 
programs*  in  lieu  of  formal  demonstration. 

4.1.2  Parts  Control 

Another  part  of  the  DoD  program  for  reliability  achievement  is  directed 
toward  parts  selection  and  control.  MIL-STD-609,  Failure  Rate  Sampling  Plan 
and  Procedure,  is  used  to  evaluate  the  failure  rate  of  parts.  MIL-STD-790, 
entitled  "Reliability  Assurance  Program  for  Electronic  Parts  Specification" 
details  actions  needed  to  assure  reliability  achievement  in  component  parts. 
Additionally, the  Defense  Electronic  Supply  Center  (DESC)  maintains  a stock  of 
selected  standard  parts  which  may  be  acquired  by  vendors  for  construction  of 
DoD  systems. 

4.1.3  Quality  Conformance 

Table  A-2  highlights  some  of  the  major  quality  specifications  now  being 
used  to  assure  conformance  to  specification.  These  include  the  sampling 
inspection  procedures,  environmental  test  methods,  and  quality  control  pro- 
gram requirements. 

The  DoD  also  maintains  a wide  range  of  specifications  and  standards 
on  related  topics  such  as  system  safety,  configuration  control,  etc.,  to 
guide  the  system  developer.  Recently  within  the  DoD,  some  individuals  have 
expressed  the  opinion  that  too  many  specifications  are  being  used,  leading 
to  over-control  of  the  design  and  production  process;  an  effort  is  being 
made  to  liberalize  quality  control  requirements. 


*A  bum-in  program  typically  requires  that  the  equipment  operate  failure- 
free  for  some  period  ranging  between  25  and  100  hours,  a value  generally 
much  less  than  the  expected  MTBF.  If  the  equipment  fails,  the  equipment 
is  repaired  and  the  test  is  repeated  until  it  achieves  the  required 
failure-free  operation. 


A-12 


Table  A-l.  DOD  RELIABILITY  AND  MAINTAINABILITY  SPECIFICATIONS 


Reference 

Title 

Essential  Elements 

MIL-HDBK-217B 

Reliability  Prediction  of 
Electronic  Equipment 

Provides  standard  methods  for  performing  parts-stress  and 
parts-count  prediction  along  with  supporting  data  for  a wide 
range  of  commonly  used  electronic  and  electromechanical  parts. 
These  techniques  are  applicable  during  system  design  prior  to 
equipment  fabrication  and  test. 

MIL-STD-470 

Maintainability  Program 
Requirements 

Defines  the  maintainability  program  requirements  to  be  accom- 
plished by  the  system  contractor , including  activities  such 
as  (1)  program  plan,  (2)  maintenance  concept,  (3)  design 
criteria,  (4)  design  trade-off,  (5)  vendor  control,  (6)  pro- 
gram integration,  (7)  design  review,  (8)  data  collection, 

(9)  demonstration  test,  and  (10)  program  reporting. 

MI L- STD- 471A 

Maintainability  Verification, 
Demonstration,  or  Evaluation 

Establishes  uniform  test  methods  and  procedures  for  assessing 
system  maintainability  parameters  such  as  maintenance  man- 
hours, and  mean,  median,  and  critical  percentile  downtime. 

MIL-HDBK-472 

Maintainability  Prediction 

Provides  four  techniques  to  estimate  the  expected  maintain- 
ability characteristics  of  a system  during  the  design  phase. 

The  technique  includes:  (1)  a system  time  element  synthesis 
technique,  (2)  a part-time  element  synthesis  technique,  (3)  a 
task  assessment  technique  which  relates  task  characteristics 
to  maintenance  time,  (4)  a maintenance  time  combination  model. 

MIL-STD-690B 

Failure  Rate  Sampling  Plans 
and  Procedures 

Provides  procedures  for  failure  rate  qualification , and  a 
sampling  plan  based  on  exfonential  distributions. 

MIL- STD- 72 IB 

Definition  of  Effectiveness 
Terms  for  Reliability,  Main- 
tainability, Human  Factors, 
and  Safety 

Contains  the  definitions  of  parameters  and  terms  used  in 
performance  assurance. 

MIL-STD- 756A 

Reliability  Prediction 

Establishes  uniform  procedure  for  predicting  the  quantitative 
reliability  of  DoD  systems.  Provides  gross  active-element- 
prediction  procedure  and  cites  techniques  in  MIL-HDBK-217  for 
detailed  analysis. 

MIL-STD-757 

Reliability  Evaluation  from 
Demonstration  Data 

Provides  uniform  procedures  for  evaluating  achieved  reliabil- 
ity and  details  the  need  for  reliability  block  diagram,  mis- 
sion evaluation,  reliability  calculations,  etc. 

MIL- STD- 7 8 1 B 

Reliability  Tests:  Exponen- 
tial Distribution 

Outlines  test  levels  and  test  plan  for  reliability  qualifica- 
tion (demonstration) , production  acceptance,  and  for  longevity 
tests.  The  test  plans  are  applicable  to  systems  for  which 
failure  patterns  follow  the  exponential  or  Poisson  distribu- 
tion. These  tests  do  not  replace  design,  performance, 
environmental,  or  other  design-development  tests. 

MIL- STD-785 

Reliability  Programs  for  Sys- 
tems and  Equipment  Development 
and  Production 

This  standard  establishes  the  criteria  for  the  preparation  and 
implementation  of  a reliability  program  plan  by  system  con- 
tractors. Key  elements  of  the  program  requirements  include: 

(1)  reliability  organization,  (2)  program  interface  with  other 
performance  activities,  (3)  supplier  control  program,  4) 
program  review,  (5)  design  analysis,  (6)  reliability  analysis, 

(7)  parts  reliability,  (8)  failure  mode  and  effects  analysis, 

(9)  critical  item  control,  (10)  design  review,  (11)  test  plan, 

(12)  development  testing,  (13)  demonstration  testing,  (14) 
failure  data,  (15)  production  reliability  control,  (16)  repro- 
curement control,  and  (17)  reporting. 

Table  A-2 . MAJOR  DOD  QUALITY  SPECIFICATIONS 


Reference  Title  Essential  Elements 

MIL-STD-105  Sampling  Procedure  Establishes  sampling  plans  and 
and  Tables  for  procedures  for  inspection  by 

Inspection  by  attributes,  by  which  the  unit 

Attributes  of  production  is  classified 

simply  as  defective  or  non- 
defective by  some  established 
criteria . 

MIL-STD-810C  Environmental  Establishes  uniform  environ- 

Test  Methods  mental  test  methods  for 

determining  the  resistance  of 
equipment  to  the  effects  of 
natural  and  induced  environ- 
ments. The  document  describes 
test  conditions,  test  proce- 
dures, test  facilities  and 
apparatus,  test  sequence, and 
detailed  test  methods. 

MIL-Q-9858  Quality  Program  Sets  forth  the  requirements 

Requirements  for  a system  supplier's 

quality  control  program. 

Major  elements  include: 

(1)  Organization 

(2)  Quality  planning 

(3)  Work  instruction 

(4)  Records 

(5)  Corrective  action 

(6)  Cost  related  to  quality 

(7)  Drawing,  documentation, 
and  changes 

(8)  Measuring  and  test 
equipment 

(9)  Production  testing 

(10)  Inspection  equipment 

(11)  Metrology  requirement 

(12)  Control  of  purchase 

(13)  Manufacturer  control 

(14)  Contractor-government 
coordination 


A-14 


I 


4.1.4  Warrant 


Z 

On  a limited  basis,  some  procurements  have  made  use  of  reliability  im- 
provement warranties  (RIW) . The  RIW  is  a long-term  agreement  '3-5  years) 
requiring  the  contractor  to  repair  all  items  which  fail  durir  the  period 
under  a firm  fixed-price  contract.  Any  action  which  the  cont  actor  can  take 
to  either  reduce  the  number  of  failures  or  lower  the  time  to  repair  will 
increase  his  profit  margin,  motivating  him  to  improve  the  equipment  by  build- 
ing in  reliability  during  design  and  through  the  development  and  implementa- 
tion of  "no-cost"  (to  the  government)  engineering  changes.  Preliminary 
results  indicate  the  RIW  concept  to  be  effective. 

4 . 2 Operational  Assessment 

Complementary  to  the  acquisition  control  techniques,  the  DoD  has  re- 
quirements for  operational  test  and  evaluation  of  newly  developed  systems 
prior  to  wide-scale  deployment.  DoD  Directive  5000.3,  Test  and  Evaluation, 
(Key  Document  A- 5)  sets  forth  the  general  policy  for  such  testing.  These 
tests  are  made  by  the  services'  own  people  to  ascertain  suitability  of  the 
system  to  the  full  service  environment.  The  services  maintain  test  groups 
separate  from  the  development  agency  to  provide  an  unbiased  assessment. 
Although  the  groups  gather  some  numerical  data,  their  tests  are  generally 
more  qualitative  than  quantitative. 

After  field  deployment  of  the  system,  the  services  collect  failure  and 
maintenance  data,  e.g.,  Air  Force  AFM-66-1,  Navy  3M.  These  data  are  analyzed 
by  cognizant  service  engineering  groups.  If  repetitive  failure  patterns  are 
observed,  action  is  taken  to  identify  the  cause  and  to  establish  corrective 
actions,  leading  to  modification  of  the  field  units  if  warranted.  If  the 
item  is  still  in  production, the  action  will  be  coordinated  with  the  develop- 
ing agency,  which  will  in  turn  advise  the  contractor  to  modify  his  production 
units. 

Field  performance  data  are  also  used  to  establish  the  reliability  and 
maintainability  goals  or  requirements  for  future  system  developments. 

4 . 3 Developing  Trends 

As  noted,  the  emphasis  within  the  DoD  is  changing  from  a total  demand 
for  technical  performance  to  a better  balance  between  technical  performance 
and  cost.  Some  reliability  assurance  efforts  tended  to  emphasize  organiza- 
tion structure,  documentation,  and  analysis  judged  to  have  little  or  no 
effect  on  designs.  Fxpensive  demonstration  tests  were  often  performed  so 
late  that  they  had  little  opportunity  to  affect  design.  This  has  led  to  a 
reexamination  of  the  methods  used  and  their  effect  on  the  product's  reli- 
ability and  maintainability. 
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As  part  of  this  new  trend  the  Naval  Material  Command  has  outlined  the 

following  new  policy: 

Objectives 


• Improve  fleet  readiness 

• Minimize  life-cycle  cost 

Acquisition  Fundamentals 

• Contract  for  reliability 

*•  Requirements  not  goals  (requirements  may  be  nonquantitative) 
••  Incentives  for  reliable  design 
*•  Reliability  in  source  selection 
••  Life-cycle  cost  consideration 


Design  to  minimize  failure 

••  Mission  and  environmental  profiles 
• • Design  alternatives  studies 
••  Numerical  allocation 
• • Conservative  derating  criteria 
• • Stress  analysis 
••  Sneak-circuit  analysis* 

• • Worst-case  tolerance  analysis 
••  Failure  modes  and  effects  analysis 
••  Parts  and  materials  selection  and  screening 
• • Design  reviews 


• Integrate  testing  to  verify  design 

••  Mission  profile  development  test  (TAAF) 

• • Design  limit  qualification  test 
• • Mission  profile  demonstration  test 

••  Failure-free  random  vibration  acceptance  (electronics) 
• • Failure-free  all  equipment  screening 


*An  analysis  of  interactions  between  components,  modules,  and  subsystems. 
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• Prevent  failure  recurrence 
••  Failure  reportinq 

••  Failure  analysis 
••  Corrective  action 

• Sustain  reliability  in  production 
• • Quality  assurance 

• • Process  controls 

••  Acceptance  testing  and  inspection 

• Sustain  reliability  in  service  use 
• • Initial  fleet  tracking 

••  Contractor  corrective  action  responsibility 


Impact 

• Reduce  maintenance  and  support  burden 

• Increase  certainty  of  reliable  material  acquisition 

• Strengthen  Navy-contractor  technical  team 

This  concept  is  documented  in  the  Naval  Material  Command's  directive 
NAVMATINST  3000. 1A. 

The  DoD  is  in  the  process  of  issuing  a comparable  document  emphasizing 
reliability  design  analysis  and  development  testing  to  achieve  reliability 
growth*  during  the  development  process.  More  emphasis  will  be  placed  on 
assessment  than  on  demonstration  to  prescribed  levels  of  producer  and  con- 
sumer risks.  It  is  expected  that  broader  use  will  be  made  of  warranties  as 
an  adjunct  to  the  performance  assurance  program  as  a means  of  motivating  the 
contractor  to  achieve  reliability  and  maintainability  growth. 

Also  pending  within  the  DoD  are  specifications  addressing  reliability 
assurance  as  it  relates  to  computer  software  and  human  factors. 

4 . 4 Lead  Responsibility  and  Interfaces 


The  overall  policy  and  procedures  are  established  at  the  DoD  level. 
Each  service  may  use  the  procedures  directly  as  in  the  case  of  MIL-STDs  or 
develop  implementing  procedures  as  in  the  case  of  directives,  e.g.,  DoD 
Directives  5000.1  and  5000.2.  These  instructions  flow  down  separately 


♦Reliability  growth  refers  to  the  rate  at  which  reliability  improves.  A 
considerable  body  of  literature  has  reported  ways  to  increase  this  rate. 
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through  the  various  organizational  channels,  i.e.,  system  acquisition  com- 
mand, operations,  and  logistics  activities.  Within  each  of  these  activities 
the  procedures  are  invoked  as  required  in  the  acquisition,  operation,  and 
support  of  systems. 

The  key  interfaces  are  between  the  DoD  elements  and  their  major  contrac- 
tors and  suppliers.  The  wide  range  of  DoD  specifications  define  the  nature 
of  this  interface.  Within  the  government,  the  DoD  has  limited  interface 
with  other  government  agencies  with  common  points  of  interest  on  selected 
systems , e . g . , FAA , NASA , UMTA . 


PROGRAM  COST  AND  EFFECTIVENESS 


5.1  Program  Cost 


The  added  cost  of  DoD  performance  assurance  activities  is  difficult  to 
determine  due  to  its  close  association  with  normal  engineering  and  test 
activities,  but  several  studies  have  attempted  to  show  the  cost-effectiveness 
of  these  activities.  Although  the  results  of  the  studies  supported  the  use 
of  performance  assurance,  the  limited  scope  of  the  investigations  prevent 
them  from  being  conclusive.  DoD  argues  that  without  these  efforts  complex 
systems  would  have  insufficient  reliability  to  meet  mission  requirements. 

The  effect  of  reliability  on  system  life-cycle  cost  can  be  clearly  shown; 
the  analysis  indicates  it  to  be,  in  many  cases,  a major  cost  driver  which 
must  be  controlled.  The  missing  element  preventing  a true  effectiveness 
assessment  is  the  total  cost  of  the  assurance  program. 

DoD  equipment  suppliers  estimate  that  a full  application  of  MIL-STD- 
781 , -785, -470,  and -471  could  add  10  percent  to  the  cost  of  a prototype  system. 
More  common  actual  expenditures  would  probably  be  on  the  order  of  1 to  3 
percent.  It  is  important  to  note  that  these  development  costs  can  be  small 
for  each  unit  if  they  are  spread  across  a large  production  base.  Use  of 
high-reliability  parts  can  further  increase  both  development  and  production 
costs. 

The  cost  of  a full  MIL  specification  program  is  not  known.  One  manu- 
facturer who  built  similar  products  for  both  DoD  and  industry  estimated  that 
the  DoD  methods  added  60  percent  to  the  production  cost  of  the  DoD  item 
over  that  of  the  unit  manufactured  to  commercial  specifications.  He  also 
stated  that  the  quality  and  reliability  of  the  commercial  product  equaled 
or  exceeded  that  of  the  military  product.  The  aviation  community  requires 
good  quality,  but  is  less  demanding  in  regards  to  design  details  and  asso- 
ciated documentation . 

5.2  Program  Effectiveness 

Overall,  the  DoD  performance  assurance  program  has  to  be  classed  as  a 
success.  The  program  has  produced  weapon  systems  with  acceptable  reliability 
and  maintainability  levels.  Had  the  failure  rates  experienced  by  DoD  systems 
during  the  early  1950s  continued,  the  highly  sophisticated  weapon  systems 
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now  being  used  would  be  totally  impractical.  Programs  which  place  emphasis 
on  a clear  statement  of  system  requirements  and  follow  with  comprehensive 
design  analysis  and  testing  appear  to  be  the  most  effective  in  improving 
the  availability,  reliability,  and  maintainability  of  DoD  systems. 
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PERFORMANCE  ASSURANCE  AS  PRACTICED  BY  THE 
NATIONAL  AERONAUTICS  AND  SPACE  ADMINISTRATION 


1.  HISTORICAL  DEVELOPMENT  AND  TRENDS 

The  NASA  performance  assurance  program,  consisting  of  two  components, 
quality  assurance  and  reliability  assurance,  evolved  directly  from  DoD 
practices  of  about  1959-63. 

The  period  from  1952  to  the  early  1960s  was  characterized  by  rapid 
technological  change  and  attempts  by  the  military  to  realize  the  benefits 
of  a vast  array  of  promising  new  technological  developments  as  fast  as 
possible.  The  size,  complexity,  specialization, and  scope-of-deployment  of 
aerospace  systems  increased  rapidly  during  this  era,  thereby  stimulating 
an  ever-increasing  emphasis  on  reliability  assurance  to  overcome  the  low 
reliability  of  these  new  systems.  As  General  Simon*  points  out,  the  dis- 
tinction between  reliability,  R,  and  very  high  reliability  (VFR)  was  not 
an  important  distinction  at  that  time  because  the  reliability  was  so  low. 
Most  programs  were  geared  to  achieving  comparatively  low  reliability  goals, 
low  enough  to  allow  the  application  of  rigorous  sampling,  testing, and 
operational  verification  techniques.  Thus,  this  era  was  characterized 
by  the  rapid  development  of  quantitative  methods  for  predicting,  testing, 
and  demonstration  reliability.  By  1964  approximately  30  to  43  percent  of 
government  contracts  contained  quantitative  reliability  requirements.** 

The  NASA  era  began  in  the  early  1960s  with  the  conversion  of  military 
guided-missile  technology  to  space  exploration  requirements.  Initially, 

NASA  also  adopted  military  reliability  and  quality  assurance  programs. 
However,  it  soon  became  apparent  that  NASA  would  have  to  devise  its  own 
reliability  and  quality  assurance  program  in  order  to  assure  the  very  high 
system  reliability  required  to  accomplish  its  primary  mission.  This  mission 
was  to  land  men  on  the  moon  under  the  watchful  eye  of  many  millions  of 
televiewers  and  assure  that  the  risk  to  astronauts  would  be  "no  greater 
than  that  accepted  by  an  average  man  pursuing  his  normal,  daily  activities". 

*L.  E.  Simon,  "The  Relation  of  Engineering  to  Very  High  Reliability", 
Proceedings  - Tenth  National  Symposium  on  Reliability  and  Quality  Control, 
1964,  pp  226. 

**E.  F.  Dertinger,  "Status  of  Reliability  Requirements  in  Government  Con- 
tracts", Proceedings  - Eleventh  National  Symposium  on  Reliability  and 
Quality  Control,  1965. 
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Therefore,  it  became  necessary  to  develop  performance  assurance  programs 
and  techniques  which  did  not  depend  upon  statistical  validation.  In  essence, 
NASA  developed  approaches  aimed  at  discovering  and  correcting  failure 
mechanisms  throughout  the  entire  system  acquisition  phase.  NASA  placed 
unprecedented  emphasis  on  reliability  analysis  during  the  design  state, 
integrated  environmental  testing  and  test-to-failure  of  components  during 
the  development  stage,  and  system  checkout  before  launch. 

2.  SCOPE  OF  NASA  PERFORMANCE  ASSURANCE  PRACTICES 

NASA's  approach  to  performance  assurance  involves:  (1)  an  extensive 
in-house  effort  which  precedes  the  acquisition  of  any  system  or  major  sub- 
system, (2)  contractor's  reliability  and  quality  programs,  (3)  continuous 
review  of  contractor's  programs  throughout  the  acquisition  process,  and  (4) 
careful  monitoring  of  system  performance  during  operation.  The  approach 
is  applied  to  launch  vehicles  and  spacecraft. 

The  preliminary  in-house  effort  involves  three  levels  of  activity  — 
a feasibility  study,  mission  planning,  and  acquisition  planning. 

2 . 1 Feasibility  Study 

First,  a feasibility  study  is  conducted,  similar  in  scope  to  that  con- 
ducted by  an  electric  utility  in  planning  for  capacity  expansion.  During 
this  phase,  the  reliability  requirements  and  factors  are  considered  rela- 
tive to  those  already  achieved  by  comparable  systems  in  operation. 

2 . 2 Mission  Planning 

Second,  the  mission  and  system  concepts  are  refined.  During  this  phase, 
the  reliability  and  quality  program  requirements  are  established.  The  em- 
phasis on  various  aspects  of  reliability  and  quality  assurance  are  tailored 
to  fit  specific  mission  requirements.  Critical  mission  parameters  are 
identified,  analyzed,  and  documented. 

2. 3 Acquisition  Planning 

The  third  stage  in  the  preliminary  in-house  effort  is  the  preparation 
of  a procurement  plan,  the  preparation  of  procurement  documents,  and  the 
evaluation  of  proposals  and  bidders.  Because  NASA  often  worked  at  or  just 
beyond  the  technical  state  of  the  art,  considerably  more  time  and  effort 
was  spent  in  specifying  and  evaluating  contractor's  reliability  and  quality 
assurance  programs  than  in  specifying,  standardizing,  and  evaluating  tech- 
nical approaches  and  hardware.  NASA  was  also  very  sensitive  to  past  relia- 
bility and  quality  performance  by  contractors.  In  most  cases,  NASA  depended 
upon  the  services  of  a prime  contractor  to  work  out  detailed  approaches  to 
meet  the  reliability  goals,  to  apportion  requirements  within  the  system,  and 
to  develop  and  negotiate  techniques  for  reliability  assurance  at  all  levels 
in  the  system  hierarchy. 
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Contractor's  Reliability  and  Quality  Programs 


NASA  performance  assurance  requirements  for  contractors  are  summarized 
in  two  documents:  "Quality  Program  Provisions  for  Aeronautical  and  Space 
System  Contractors"  (Key  Document  B-l) , and  "Reliability  Program  Provisions 
for  Aeronautical  and  Space  System  Contractors"  (Key  Document  B-2) . 

2.4.1  Quality  Program 

The  NASA- imposed  quality  assurance  program  consists  of  twelve  elements 
or  formalized  activities: 

• Quality  program  management  and  planning 

• Design  and  development  controls 

• Identification  and  data  retrieval 

• Procurement  controls 

• Fabrication  controls 

• Inspection  and  testing 

• Nonconforming  article  and  material  control 

• Metrology  controls 

• Stamp  controls* 

• Handling,  storage,  preservation,  marking,  labelling,  packaging, 
packing,  and  shipping 

• Sampling  plans,  statistical  planning,  and  analysis 

• Government  property  control 

It  is  important  to  recognize  that  NASA  contractors  often  retain  a major 
role,  not  only  during  system  acquisition  and  test,  but  during  flight  opera- 
tions as  well.  Thus,  for  example,  the  system  contractor  (prime  contractor) 
often  addresses  quality  program  requirements  as  outlined  in  Table  B-l. 

2.4.2  Reliability  Program 

The  NASA- imposed  reliability  program  consists  of  twenty  elements  or 
formalized  activities  in  three  categories: 

• Reliability  Program  Management 
• • Organization 

••  Reliability  program  plan 
• • Reliability  program  control 
••  Reliability  progress  reporting 

♦Approval  is  often  signified  by  a stamp.  Access  to  these  stamps  is  tightly 
controlled. 
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Table  B-l.  QUALITY  PROGRAM  REQUIREMENTS 

Stage 

Contractor  Responsibility 

Design  and  development 

Quality  program  plan 

Quality  criteria 

Purchasing 

Purchase-documents  control 

Source  inspection 

Material  receipt 

Process  control  and  inspection 

Qualification  and  conformance  tests 

Fabrication 

Process  control  and  inspection 

Qualification  and  conformance  tests 

System  assembly 

End-item  testing 

Flight  operations 

Data  collection 

Data  analysis 

Data  feedback 

• • Reliability  training 
• • Supplier  control 

••  Reliability  of  government  furnished  property 

• Reliability  Engineering 

• • Design  specifications 
• • Reliability  prediction 

••  Failure  mode,  effect,  and  criticality  analysis 

••  Maintainability  of  the  system  and  elimination  of  human-induced 
failure 

• • Design  review  program 

••  Problem  and  failure  reporting  and  collection 
••  Standardization  of  design  practices 
• • Parts  and  materials  program 

• Testing  and  Reliability  Evaluation 
• • Reliability  evaluation  plan 
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• • Testing 

••  Reliability  assessment 

Here  again,  the  NASA  system  contractor  (prime  contractor)  often  retains  a 
major  role  throughout  the  entire  mission. 

2. 5 Continuous  Review  During  Acguisition  and  Monitoring  of  System 
Performance 

NASA's  practice  of  establishing  and  maintaining  a close  relationship 
with  the  prime  acquisition  contractor  during  system  operation,  coupled  with 
its  imposition  of  operations  requirements  within  the  scope  of  the  system 
contractor's  quality  and  reliability  program,  lead  the  acquisition  contrac- 
tor to  have  a continuing  stake  in  the  operational  success  of  the  system. 
(This  stake  is  often  made  explicit  through  the  use  of  incentive  contracts*.) 
In  practice,  this  approach  to  performance  assurance  leads  system  suppliers 
to  take  considerable  interest  in  continuous  review  and  monitoring  throughout 
the  entire  life  of  the  system. 

As  a result,  many  contractors  develop  and  maintain  their  own  extensive 
failure  reporting  and  follow-up  systems**,  often  used  in  pre-launcht  and 
post-launch++  mission  analyses.  Acquisition  contractors  also  participate 
in  diagnosing  and  recommending  repair  actions  during  launch  and  in-space 
operations . # 


3.  APPLICABILITY  OF  NASA  METHODS  TO  THE  ELECTRIC  POWER  INDUSTRY 

Two  important  and  interdependent  aspects  of  the  NASA  approach  to  per- 
formance assurance  are  worth  considering  for  application  by  the  electric 
power  industry.  One  is  NASA's  emphasis  on  establishing  detailed  require- 
ments and  plans  for  continual  and  graduated  testing  throughout  all  phases 
of  the  system  acquisition  process.  The  other  is  NASA's  emphasis  on  under- 
standing anv.  simulating  the  operational  environment  during  tests.  NASA 
also  places  considerable  emphasis  on  reliability  assurance  during  the  design 
phase.  This  emphasis  and  the  associated  approach  are  similar  to  those 
described  as  DoD  practice  (Appendix  A) . 


*A.  J.  Moskovitz,  "NASA's  Application  of  NPC  250-1  in  Incentive  Contracts", 
Proceedings  - Eleventh  National  Symposium  on  Reliability  and  Quality 
Control , 1965 . 

**E.g.,  see  G.  S.  Gordon,  "Failure  Reporting  on  Satellite  Programs", 
Proceedings  - 1967  Annual  Symposium  on  Reliability , pp.  128. 

+E.g.,  F.  P.  Kiefer,  et  al , "Man-Rating  the  Gemini  Launch  Vehicle", 
Proceedings  - 1966  Annual  Symposium  on  Reliability , pp  260. 

++E.g.,  B.  B.  Klawans  and  E.  C.  Thomas,  "Flight  Performance  Analysis  of 
Space  Systems",  Proceedings  - 1969  Annual  Symposium  on  Reliability . 

#E.g.,  W.  R.  Abbott  and  L.  E.  Jenkins,  "Flight  Failure  Analysis", 
Proceedings  - 1969  Annual  Symposium  on  Reliability , pp  244. 
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3 . 1 Test  Requirements  and  Plans 


NASA  typically  devoted  a considerable  effort  to  develop  mission 
effectiveness  criteria  and  system  availability  requirements  during  the 
pre-acquisition  phase.  Experience  from  previous  flights  and  tests  was 
incorporated  to  identify  critical  problems  so  that  contractors  could  work 
out  and  negotiate  test  requirements  and  testing  plans  at  appropriate  levels 
of  test  emphasis. 

Electric  power  utilities  could  benefit  from  NASA's  experience  by  plac- 
ing more  emphasis  on  establishing  graduate‘s  test  requirements  during  the 
planning  phase  and  by  formulating  test  plans  in  cooperation  with  their 
contractors.  A mechanism  could  be  developed  to  make  available  these  test 
plans  and  subsequent  evaluations  of  test  results  to  all  companies. 

3.2  Simulating  the  Operational  Environment 

NASA  places  great  emphasis  on  testing  in  an  environment  approximating 
operational  conditions.  NASA  centers  and  contractors  employ  specialized 
teams  to  analyze  and  understand  the  operational  environment  to  design  appro- 
priate environmental  simulations  and  to  evaluate  the  effect  of  environmental 
stress  and  peculiarities. 

Electric  utilities  and  their  contractors  would  benefit  from  the  prepara- 
tion of  environmental  simulation  guidelines.  Further,  it  may  be  appropriate 
to  consider  establishing  national  environmental  testing  facilities  and 
specialized  support  staffs. 


4.  DESCRIPTION  OF  APPLICABLE  PRACTICES 

The  NASA  approach  to  performance  assurance  allows  enormous  flexibility. 
Each  prime  contractor  negotiates  a performance  assurance  program  in  coopera- 
tion with  the  NASA  Center  in  charge  of  the  program,  within  the  guidelines 
specified  in  Key  Documents  B-l  and  B-2.  Therefore,  rather  than  describe  a 
specific  NASA  program  in  order  to  convey  the  essence  of  NASA's  approach  to 
performance  assurance,  we  have  chosen  to  describe  a "prototypical"  NASA 
performance  assurance  program  drawn  from  actual  programs  that  have  been 
documented  in  the  most  detail.  In  constructing  this  prototypical  program 
we  have  tried  to  emphasize  those  elements  most  peculiar  to  NASA  that  also 
seem  most  applicable  for  consideration  by  the  power  industry. 

4.1  Key  Program  Elements 

4.1.1  Spacecraft  Mission  Effectiveness  Analysis  (SMEA) 

NASA's  emphasis  on  graduated  testing  in  simulated  or  actual  mission 
environment  led  to  a philosophy  of  treating  each  space  mission  as  a test 
for  the  next.  NASA  utilizes  the  SMEA  to  establish  explicit  and  quantitative 
a priori  measures  of  achievement  during  the  planning  stage  and  to  express 
these  measures  relative  to  a posteriori  evaluations  of  achievement  by 


systems  already  in  operation.  As  used  in  the  orbiting  geophysical  observa- 
tory (0G0)  program  by  NASA*,  the  achievements  expected  of  one  of  the  series 
of  "standard"  satellites  were  expressed  relative  to  the  achievements  of 
others  in  the  same  series  already  in  operation.  In  the  case  of  0G0,  the 
effectiveness  was  defined  by  an  independent**  contractor  (Planning  Research 
Corporation)  in  cooperation  with  the  ultimate  user  (NASA)  and  the  system 
acquisition  contractor  (TRW) . One  result  was  the  definition  of  an  overall 
figure  of  merit  for  the  system,  used  throughout  the  development  of  the  OGO 
series  to  postulate  and  evaluate  improvement  in  mission  performance . t 

4.1.2  Launch  Availability  Analysis 

As  utilized  by  NASA,  availability  analysis  is  performed  throughout 
the  entire  life  cycle  of  the  system  from  concept  through  operation. 

As  applied  on  Saturn  Vtt,  the  analysis  involved  a large-scale  digital 
computer  simulation  model  that  analyzed  and  simulated  the  performance  of 
the  system  during  all  operations  up  to  launch.  This  model  utilized  system 
reliability  data,  maintainability  data,  and  operating  time  constraints  to 
determine  the  probable  Saturn  V system  availability  at  launch  time. 

The  failure  rate  methodology  involves  the  transition  from  predictive 
data  to  assessment  data,  combining  failure  rate  predictions  with  field 
failure  data  through  the  use  of  both  classical  and  Bayesian  techniques. 

The  model  programs  approximately  2000  equipment  categories  used  in  over 
200  operational  events  leading  up  to  the  Saturn  V liftoff.  The  data  bank 
stores  generic  failure  rates,  maintenance  times,  environmental  factors, 
stress  levels,  and  operating  times  for  the  major  components,  subsystems, 
and  systems  within  each  individual  pre-launch  event. 

4.1.3  Flight  and  Test  Performance  Analysis 

NASA  performance  analysis  includes  preparation,  implementation,  and 
communication  phases.  The  formal  performance  analysis  described  below  was 
developed  for  application  to  the  Nimbus  Meteorological  Satellites,  the 
Applications  Technology  Satellites,  the  Geophysical  Earth  Orbiting  Satel- 
lites, and  the  biosatellite  and  several  military  systems  (the  description 
is  taken  from  the  previously  cited  paper  by  Klawans  and  Thomas) . 

During  preparation,  the  scope  of  the  test  performance  analysis  is 
projected  in  terms  of  magnitude,  schedule,  interfaces,  and  cost.  Flight 
and  test  performance  analysts  are  assigned  to  particular  subsystems  and 
trained  for  the  analysis  ahead. 


*A.  Leventhal  and  C.  E.  Bloomquist,  "Spacecraft  Mission  Effectiveness", 
Proceedings  - 1968  Annual  Symposium  on  Reliability , pp  615. 
‘♦Independent  contractors  do  not  subcontract  to  acquisition  contractors 
but  work  directly  for  the  purchasing  agency. 

+A.  Leventhal  et  al , "Spacecraft  Failure  Rates  — Where  Are  We?", 
Proceedings  - 1969  Annual  Symposium  on  Reliability , pp  444. 

++R.  A.  Venditti  and  R.  M.  Sineath,  Jr.,  "Saturn  V System  Reliability 
Analysis",  Proceedings  - 1969  Annual  Symposium  on  Reliability , pp  567. 
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The  implementation  phase  has  two  segments.  The  first  segment  is  an 
in-depth  analysis  of  anomalous  performance  of  spacecraft  during  flight  or 
test  to  isolate  the  cause  of  a failure  to  the  piece-part  level  and  det  mine 
the  extent  of  its  effect  on  the  mission.  The  second  segment  is  a total 
system  capability  determination  initiated  during  system  development  tests 
and  continually  updated  through  qualification,  acceptance,  and  flight  test- 
ing. Information  is  gathered  continually  to  define  the  adequacy  of  the 
system  test  requirements  and  the  total  performance  envelope  that  is  avail- 
able with  maximum  utilization  of  the  present  design  and  to  identify  those 
changes  that  may  be  incorporated  to  improve  performance. 

Most  contractors  use  the  flight  and  test  performance  analysis  team 
concept  to  communicate  the  results  of  the  team  effort;  frequent  working- 
level  briefings  are  held  for  all  evaluation  personnel.  In  general,  the 
company  position  on  each  observed  anomaly  is  stated  in  a scheduled  letter 
report  to  NASA.  This  letter  is  followed  by  an  in-depth  analysis  report 
giving  detailed  information  on  all  anomalies.  The  flight  and  test  perform- 
ance analysis  group  also  prepares  the  On-Orbit  User's  Handbook  which  con- 
tains systems  specifications,  systems  performance  during  acceptance  testing 
and  launch  preparation,  normal  transient  signatures  of  the  actual  hardware 
being  flown,  and  overall  system  test  results.  For  some  programs  a computer 
tape  is  prepared  in  addition  to  the  on-orbit  handbook  so  the  prior  perform- 
ance may  be  displayed  automatically  and  compared  to  current  performance. 

The  functionally  organized  flight-performance  analysis  team  provides  a 
degree  of  cross-fertilization  that  is  difficult  to  obtain  within  a totally 
project-oriented  program.  A failure  is  not  dismissed  when  the  immediate 
problem  has  been  resolved.  Its  impact  on  every  current  and  anticipated 
program  is  evaluated  and  corrective  action  is  taken  across  the  board  to 
prevent  recurrence. 

4.1.4  Defining  Reliability  Test  and  Demonstration  Requirements 

As  R.  B.  Carpenter  points  out*,  requirements  for  reliability  have  been 
increasing  by  several  orders  of  magnitude  for  eacl-)  generation  system,  while 
the  ability  to  demonstrate  reliability  performance  has  shown  a corresponding 
decrease.  If  the  system  to  be  demonstrated  actually  has  the  very  high  relia- 
bility required  by  NASA,  the  time  required  for  statistically  valid  tests  at 
the  system  level  usually  exceeds  the  time  available  for  the  whole  program. 
Hence,  NASA  programs  rarely  require  statistical  validation  at  the  system 
level.  Instead,  a series  of  integrated  tests,  combined  with  analysis,  are 
performed  as  the  system  is  designed  and  developed.  The  result  is  an  engi- 
neering confidence  level  not  definable  in  statistical  terms. 


*R.  B.  Carpenter,  Jr.,  "Demonstrating  Reliability  for  Long  Space  Missions", 
Proceedings  - Eleventh  National  Symposium  on  Reliability  and  Quality 
Control,  1965,  pp  223. 
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Optimizing  the  test  and  demonstration  program  is  a crucial  part  of 
NASA's  performance  assurance  programs.  A typical  approach,  described  by 
Carpenter  as  being  used  by  North  American  Aviation  is  based  on  the  ap- 
plication of  two  independent  analyses — pretest  analysis  and  test  emphasis 
analysis.  These  analyses,  used  in  conjunction  with  a formal  "needs" 
analysis,  provide  assurance  of  design  integrity,  even  though  statistical 
confidence  is  often  lacking. 

The  needs  analysis,  aimed  at  defining  the  types  of  tests  required  to 
evaluate  performance  of  the  system,  may  be  similar  to  those  already  de- 
scribed under  the  headings  of  spacecraft  mission  effectiveness  analysis, 
availability  analysis,  or  flight  and  test  performance  analysis.  Carpenter 
suggests  that  needs  analysis  should  be  a sequential  composite  of  the  three 
analyses . 

The  test  emphasis  analysis  (TEA)  determines  the  number  and  duration  of 
tests  required  for  individual  constituents  of  the  system.  A test  emphasis 
index  is  established  to  determine  the  sample  size  and  test  duration  for  each 
system,  component,  and  part.  Also, TEA  may  be  used  to  establish  the  percent 
of  the  budget  to  be  apportioned  to  various  levels  as  a function  of  overall 
cost  or  schedule.  The  purpose  is  not  to  demonstrate  performance  with  mathe- 
matical rigor  but  to  budget  test  effort  within  the  practical  constraints 
imposed  at  each  stage  in  the  acquisition  process. 

4.1.5  Design  Proof  Tests* 

The  design  proof  tests  used  on  Apollo  followed  MIL-E-5272  except  that 
environments  were  at  the  anticipated  maximum  level  of  stress  for  a typical 
Apollo  mission.  Each  environment  was  applied  at  the  component  level  in 
sequence  and  in  ascending  order  of  severity  to  verify  ability  to  perform 
under  single  worst-case  conditions.  Test  safety  margins  were  set  at  about 
1.33  times  the  anticipated  level  in  more  than  90  percent  of  the  possible 
situations.  Test  safety  margins  were  set  somewhere  between  the  level  of 
the  designer's  margin  and  the  normally  anticipated  level. 

4.1.6  Off-Limit  Tests 

In  the  off-limit  tests  (described  by  Carpenter)  a given  stress  is 
increased  in  small  increments  until  failure  occurs  or  until  the  design 
margin  is  exceeded  substantially.  The  purpose  of  these  tests  is  to  sup- 
plement other  test  data,  determine  comparative  reliability  between  com- 
ponent types,  increase  the  effective  sample  size  for  increased  statistical 
confidence,  and  provide  a basis  for  trade-off  analyses. 


*R.  B.  Carpenter,  Jr.,  "Apollo  Reliability  by  Demonstration  or  Assessment", 
Proceedings  - Tenth  National  Symposium  on  Reliability  and  Quality  Control, 
1964,  pp  517. 
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4.1.7  Mission  Simulation  Life  Tests 


For  application  to  Apollo,  Carpenter  reported  that  four  simulated 
"mission  cycles"  were  imposed  on  each  system  designated  for  the  test.  The 
mission  simulation  includes  all  passive  and  active  states  as  well  as  pas- 
sive and  active  environments,  both  singly  and  in  combinations.  Two  cycles 
per  system  had  to  be  accomplished  without  a failure  for  successful  comple- 
tion of  the  test  series. 

4.1.8  Acceptance 

NASA's  decision  to  buy  or  not  to  buy  was  not  made  on  the  basis  of  any 
single  test,  but  involved  an  evaluation  of  all  tests  required  during  design, 
pre-production,  production,  and  post-production  phases.  For  example*,  in 
accepting  launch  vehicles,  NASA  required  successful  completion  of  initial 
acceptance  tests  (laboratory  static  bench-test) , pre-production  tests  in 
appropriate  environments,  production  acceptance  tests  applied  at  end-use 
operating  environmental  stress  levels,  periodic  reevaluation  tests  of 
questionable  items  in  adverse  environments,  off-limit  tests,  and  extended 
time  tests. 

4.1.9  Lead  Responsibilities 

NASA  Headquarters  has  lead  responsibility  for  preparing  and  updating 
general  performance  assurance  guidelines.  These  are  implemented  by  the 
NASA  centers,  to  the  extent  that  each  center  finds  them  pertinent.  NASA 
Headquarters  reviews  the  center  programs  annually  and  submits  recommenda- 
tions for  improvement.  Although  the  centers  delegate  considerable  respon- 
sibility to  integration  contractors  and  prime  contractors,  center  personnel 
retain  responsibility  for  accepting  or  rejecting  test  plans  and  results. 

4.1.10  Government- Industry  Interface 

Because  NASA  funds  the  aerospace  industry  directly,  the  government- 
industry  interface  is  a contractual  one.  It  is  important  to  NASA  that  the 
contractual  relationship  with  contractors  employed  during  the  system  acqui- 
sition stage  be  maintained  during  system  operations. 


5.  COST  AND  EFFECTIVENESS  CONSIDERATIONS 

NASA  estimates  that  the  average  cost  of  applying  formal  performance 
assurance  to  all  launch  vehicles  and  spacecraft  during  the  1960s  was  10 
percent  of  the  system  acquisition  cost. 


*C . C.  Campbell,  "High  Reliability  for  Space  Launch  Vehicles",  Proceedings 
- Eleventh  National  Symposium  on  Reliability  and  Quality  Control,  1965, 
pp  439. 
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5.1  Program  Initiation  Costs 


Initiation  costs  were  predominantly  those  associated  with  the  develop- 
ment and  installation  of  major  test  and  mission  simulation  facilities.  A 
considerable  portion  (10  to  20  percent)  of  NASA's  total  budget  ($3-  $5 
billion  per  year)  was  allocated  to  the  installation  of  these  facilities 
during  the  first  five  years  of  NASA's  history. 

5.2  Program  Operating  Costs 

The  cost  of  operating  the  performance  assurance  program  in  its  mature 
state  (about  1965)  was  reduced  substantially*  as  launch  vehicles,  spacecraft, 
and  procedures  became  more  standardized. 

5. 3 Program  Effectiveness 

The  early  NASA  missions  suffered  from  poor  system  reliability,  evi- 
denced by  numerous  flight  delays  and  aborted  missions.  As  emphasis  was 
placed  on  performance  assurance,  the  mission  success  rate  improved.  NASA's 
basic  approach  of  identifying  mission  environmental  requirements  and  employ- 
ing exhaustive  testing  against  these  requirements  has  led  to  excellent  sys- 
tem reliability.  The  recent  Mars  landing  with  the  Viking  spacecraft  is  a 
clear  indication  of  this  achievement. 


APPENDIX  C 


PERFORMANCE  ASSURANCE  AS  PRACTICED  BY  THE 
FEDERAL  AVIATION  ADMINISTRATION  AND  AIRLINE  INDUSTRY 


1.  HISTORICAL  DEVELOPMENT 

The  Federal  Aviation  Administration  (FAA)  now  resides  within  the  De- 
partment of  Transportation,  having  at  one  time  been  an  independent  agency. 
Performance  assurance  has  long  been  a key  part  of  FAA's  concern,  particu- 
larly as  it  relates  to  airworthiness,  quality,  and  safety. 

Following  DoD's  lean  in  the  development  of  complex  systems,  the  air 
transport  industry  made  use  of  many  of  the  military-developed  systems  (e.g., 
aircraft,  radar,  navigation,  communication)  and  experienced  similar  reli- 
ability problems.  One  of  the  earliest  reliability  studies  was  triggered 
by  the  unacceptable  reliability  of  an  airborne  communications  transceiver 
in  use  by  the  commercial  airlines.  The  work  on  this  problem  performed  by 
Aeronautical  Radio,  Inc.,  (ARINC)  led  to  the  development  of  a line  of  "high 
reliability"  vacuum  tubes  for  use  in  the  equipment.  The  success  of  the 
program  led  to  ARINC' s being  requested  to  initiate  similar  efforts  for  the 
military  in  1951  — one  of  DoD's  earliest  attacks  on  the  reliability  problem. 

Through  the  years,  the  FAA  has  placed  heavy  emphasis  on  qualification 
testing.  However,  following  the  lead  of  DoD  and  NASA,  the  agency  has  ex- 
panded the  scope  of  its  reliability  and  maintainability  programs,  making 
use  of  specifications  and  procedures  developed  by  other  agencies  as 
appropriate . 


2.  SCOPE  OF  CURRENT  PRACTICES 

The  FAA  activities  related  to  performance  assurance  are  quite  broad  but 
for  discussion  purposes  may  be  categorized  to  fall  within  the  major  areas  of 
aircraft  certification,  system  acquisition  and  operation,  and  safety  programs. 

2.1  Aircraft  Certification 


FAA  ensures  that  new  aircraft,  engines  and  propellers,  and  the  compo- 
nent, parts,  and  appliances  that  go  with  them,  are  of  acceptable  quality 
when  completed,  by  requiring  that  they  be  type-certificated.  The  agency 
issues  type  certificates  for  new  models  of  aircraft,  engines,  and  propellers 
when  they  meet  prescribed  airworthiness  and  noise  standards  and  are  deemed 
safe.  In  case  of  an  approved  change  in  a type-certificated  model,  FAA 


issues  a supplementary  type  certificate,  or,  if  the  change  is  substantial 
enough  to  warrant  it,  a new  type  certificate.  Foreign  aircraft  seeking 
U.S.  certification  are  subject  to  standards  comparable  to  those  required 
to  be  met  by  U.S.  aircraft. 

2 . 2 System  Acquisition  and  Operation 

FAA  is  responsible  for  the  National  Airways  System,  a complex  of  air 
traffic  control,  navigation,  and  communications  systems  used  to  monitor 
and  control  air  movements  through  the  system.  FAA  is  responsible  for  the 
design,  acquisition,  installation,  and  operation  of  all  facets  of  the  sys- 
tem. To  meet  its  responsibility  the  administration  has  developed  a series 
of  performance  assurance  techniques  to  control  this  activity. 

2. 3 Safety  Programs 

The  FAA  also  has  a broad  spectrum  of  safety  program  activities, 
specifically  including: 

• Aircraft  Accident  and  Notification,  Investigation  and  Reporting 

• School  and  Repair  Station  Certification 

• Airport  Certification 

• The  Mechanic  Safety  Programs 

• The  Service  Difficulty  Program 

• The  Biennial  Airworthiness  Review  Program 

• The  Biennial  Operations  Review  Program 

• Flight  Inspection 


3.  APPLICABILITY  OF  FAA  METHODS  TO  THE  ELECTRIC  POWER  INDUSTRY 

3.1  System  Acquisition  and  Operation  Techniques 

The  system  acquisition  and  operation  techniques  employed  by  the  FAA 
to  a large  extent  are  patterned  after  those  of  the  DoD  and  NASA.  In  fact, 
FAA  has  used  many  of  the  actual  DoD  specifications  and  procedures. 

3.2  Certification  Programs 

The  FAA  certification  program  governing  equipment  owned  and  operated 
by  the  air  carriers  is  somewhat  analogous  to  the  Nuclear  Regulatory  Com- 
mission (NRC)  licensing  process.  There  are  several  techniques  within  this 
program  fostered  by  the  FAA  and  the  airline  community  worth  further  examj  na- 
tion. These  include  (1)  functional  standards  and  (2)  the  maintenance 
reliability  program. 
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3.2.1  Form,  Fit,  and  Function  Standards 

For  selected  components,  the  airline  community  develops  functional 
standards  which  provide  electrical,  mechanical,  and  functional  interchange- 
ability  among  suppliers.  In  addition  to  providing  a desirable  competitive 
environment  for  the  user,  the  concept  has  resulted  in  highly  reliable  sys- 
tem components.  Through  the  standards  program, a particular  subsystem  built 
to  the  functional  specifications  prescribed  by  the  airline  community  will 
be  used  in  most  air  carrier  aircraft  and  may  be  used  in  many  general  avia- 
tion aircraft.  This  broad  production  base,  which  may  be  shared  by  several 
vendors,  permits  the  designs  to  mature,  leading  to  the  high  reliability 
cited.  Competition  and  use  of  warranties  provide  further  mechanisms  for 
product  quality  improvement.  The  FAA  enters  into  the  process  by  its  im- 
position of  certification  requirements  on  top  of  the  basic  functional 
requirements.  It  would  appear  that  the  specification  of  form,  fit,  and 
function  would  have  direct  applicability  to  the  electric  power  industry 
for  many  common  items  used  in  power  generation. 

3.2.2  Operational  Maintenance  Reliability  Program 

As  an  extension  of  its  certification  process,  the  FAA  requires  the 
airlines  to  comply  with  established  maintenance  and  inspection  programs. 

To  do  so,  the  airlines  have  developed  a flexible  maintenance  approach 
coupled  with  a tight  reliability  control  program  that  monitors  the  failure 
performance  of  critical  aircraft  components  in  response  to  changes  in 
maintenance  policies.  The  establishment  of  a comparable  program  within 
the  electric  power  systems  for  critical  systems  could  prove  to  be  highly 
cost-effective  while  providing  the  needed  control  of  operational  systems. 

3.2.3  Safety  Programs 

The  safety  programs  developed  by  the  FAA  fit  the  unique  operational 
requirements  of  its  area  of  resp>.  isibility.  They  have  limited  application 
in  the  power  industry,  which  has  already  placed  considerable  emphasis  on 
safety. 

4.  DESCRIPTION  OF  APPLICABLE  PRACTICES 

The  FAA's  certification  programs  and  acquisition  concepts  merit  further 
discussion.  Within  the  certification  program  the  airline  procurement  con- 
cepts and  operational  assessment  procedures  are  of  particular  interest. 

4 . 1 Certification  Program 

4.1.1  System  Acquisition 

As  a partial  result  of  the  FAA  responsibilities  for  certificating  the 
quality  and  airworthiness  of  systems  acquired  by  the  aviation  industry,  a 
unique  procurement  concept  has  been  developed  within  the  airline  community. 


4. 1.1.1  Airline  Procurement  History  and  Evolution 


The  airline  procurement  process  has  developed  through  a number  of 
evolutionary  steps.  Beginning  in  the  late  1930s,  the  airlines  centralized 
the  preparation  of  avionic  equipment  specifications  and  procurements  in  a 
single  organization,  ARINC.  It  was  believed  that  if  ARINC  could  coordinate 
the  development  activities  and  equipment  needs,  standardization  would  be 
achieved  and  significant  savings  realized. 

During  the  1940s,  the  development  of  specifications  and  procurement 
techniques  took  a significant  turn.  The  preparation  of  specifications  was 
centered  in  a specific  group  of  airline  people,  known  today  as  the  Airlines 
Electronic  Engineering  Committee,  rather  than  left  to  whoever  attended  the 
general  industry  meetings  of  ARINC. 

Another  turning  point  in  the  evolution  occurred  in  the  early  1950s, 
when  the  writing  of  ARINC  Characteristics  became  a public  process  with  the 
participation  of  manufacturers.  Interchangeability  was  established  as  the 
first-priority  item,  and  this  became  the  major  purpose  of  ARINC  Character- 
istics. The  premise  was  that  "form,  fit,  and  function"  should  be  a basic 
standard  and  that  extra  operational  performance,  special  features,  and 
flexibility  should  rightfully  remain  optional  items,  to  be  purchased  by 
those  who  needed  and  could  afford  them. 

4. 1.1. 2 Airline  Procurement  Participants 

The  commercial  airline  procurement  method  involves  the  customer  (air- 
line companies) , the  supplier  (equipment  vendors) , and  several  organizations 
unique  to  the  United  States  air  transport  industry.  These  organizations 
include  the  following: 

• Airlines  Electronic  Engineering  Committee  (AEEC) 

• Avionics  Maintenance  Conference  (AMC) 

• Radio  Technical  Commission  for  Avionics  (RTCA) 

• Air  Transport  Association  of  America  (ATA) 

• Federal  Aviation  Administration  (FAA) 

The  membership,  key  activities,  and  contributions  of  each  of  these 
organizations  are  discussed  in  the  following  subsections. 

Airline  Companies.  The  U.S.  air  carriers  currently  comprise  eleven 
trunk,  nine  local  service,  and  three  cargo  carriers.  These  companies  own 
approximately  2,500  aircraft  varying  from  single-engine,  piston-powered 
vehicles  to  four-engine  jets.  It  has  been  estimated  thav  the  world's 
airlines  spend  more  than  $300  million  annually  on  avionics. 

Avionics  Vendors.  There  are  approximately  30  U.S.  manufacturers  of 
avionics  that  serve  not  only  the  air  carriers'  fleet  of  2,500  aircraft  but 
also  general  aviation  and  the  military.  The  present  general  aviation  fleet 
consists  of  about  161,500  aircraft  and  the  military  approximately  20,000 
aircraft. 
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ARINC.  ARINC  was  organ' zed  by  the  airlines  on  December  2,  1969,  to 
serve  as  the  single  licensee  and  coordinator  of  aeronautical  radio  communi- 
cations outside  the  government.  The  Airlines  Electronic  Engineering  Com- 
mittee (AEEC)  within  ARINC  is  the  focal  point  for  common  airline  avionic 
acquisition  activity. 

The  primary  function  of  the  AEEC  is  to  formulate  ARINC  Characteristics, 
form,  fit,  and  function  standards  for  electronic  equipment  and  systems. 

An  ARINC  Characteristic  has  a twofold  purpose: 

• To  communicate  to  prospective  manufacturers  of  airline  electronic 
equipment  the  general  desires  of  the  airline  technical  people, 
coordinated  on  an  industry  basis,  concerning  a particular  type 

of  equipment 

• To  promote  maximum  possible  interchangeability  without  seriously 
hampering  design  initiative 

Before  they  are  published,  these  characteristics  are  coordinated  and  ap- 
proved after  sometimes  extended  discussions  among  the  AEEC  participants. 

The  characteristics  do  not  precisely  define  the  contents  of  the  "black 
box"  but  describe  the  signals  that  enter  and  leave  the  box , and  the 
electrical,  mechanical,  and  environmental  interfaces. 

RTCA . The  Radio  Technical  Commission  for  Avionics  (RTCA)  was  formed 
as  an  association  of  more  than  100  aeronautical  organizations  of  the  United 
States.  Its  present  membership  includes  all  military  departments  and  the 
Departments  of  State,  Commerce,  and  Transportation  (FAA) , the  Federal  Com- 
munications Commission,  the  National  Aeronautics  and  Space  Administration, 
the  Air  Transport  Association,  Aircraft  Owners  and  Pilots  Association,  Air- 
line Pilots  Association,  and  manufacturing  industry  organizations.  A key 
RTCA  activity  is  the  preparation  of  documents  that  provide  minimum  perform- 
ance standards  (MPS)  and  test  procedures,  environmental  test  procedures,  and 
operational  and  technical  characteristics  of  aviation  electronics  and  tele- 
communications. These  documents  are  used  to  guide  the  preparation  of  ARINC 
Characteristics,  are  used  as  guidelines  by  manufacturers,  and  often  serve 
as  the  minimum-performance  test  criteria  of  equipment  authorized  for  use 
on  civil  aircraft  by  the  FAA  through  its  Technical  Standard  Order  (TSO) 
authorization  process. 

ATA.  The  Air  Transport  Association  of  America  (ATA)  is  a cooperative, 
non-profit  trade  and  service  organization  of  the  U.S.  scheduled  airlines. 
Through  its  member  airlines,  the  ATA  works  to  improve  airline  safety,  serv- 
ice, and  efficiency.  It  is  currently  divided  into  eight  departments,  each 
of  which  parallels  a function  of  the  airlines . ATA  activities  that  directly 
affect  airline  procurements  are  carried  out  through  a system  of  councils 
and  related  committees  made  up  of  airline  and  ATA  representatives. 

The  ATA  publishes  several  documents  that  significantly  affect  the  air- 
line procurement  process.  One  is  the  World  Airline  Suppliers'  Guide,  which 
establishes  the  policies  and  objectives  of  the  air  transport  industry  with 
respect  to  the  suppliers'  support  of  the  world  airlines'  fleet.  This  guide 
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provides  the  consensus  of  the  member  airlines  concerning  general  terms  and 
agreements,  initial  provisioning,  inventory  policies,  pricing,  value  analy- 
sis, order  administration,  packaging  and  shipping,  invoicing  warranties, 
simulators,  and  manufacturers'  technical  data. 

The  ATA  also  publishes  four  specifications  commonly  used  by  all  air- 
nes  in  their  procurement  documents  to  ensure  proper  support  from  each 
supplier  of  aircraft  products: 

• ATA  Specification  No.  100  - Manufacturers'  Technical  Data 

• ATA  Specification  No.  101  - Ground  Equipment  Technical  Data 

• ATA  Specification  No.  200  - Integrated  Data  Processing  Supply 

• ATA  Specification  No.  300  - Packaging  of  Airline  Supplies 

These  specifications  were  developed  to  provide  guidelines  to  an  in- 
creasing number  of  inexperienced  suppliers  of  the  airline  industry  and  to 
permit  mutual  savings  in  technical-data  preparation,  spare-parts  provision- 
ing, and  packaging. 

FAA.  The  FAA  role  in  the  airline  acquisition  process  is  to  certify 
the  air-worthiness  of  aircraft  and  their  equipment.  Once  a manufacturer 
has  completed  the  design  and  production  of  an  avionics  product  (which  may 
be  based  on  an  ARINC  Characteristic) , he  must  obtain  authorization  from  the 
Federal  Aviation  Administration  before  the  product  can  be  used  on  civil 
aircraft.  The  authorization  is  issued  on  the  basis  of  the  manufacturer's 
conformance  with  FAA  Regulation,  Volume  II,  Part  37,  and  the  applicable 
Technical  Standard  Order  (TSO) . 

Technical  Standard  Orders  contain  the  minimum  performance  and  quality- 
control  standards  for  products  used  on  aircraft.  The  performance  standards 
in  each  TSO  ensure  that  the  product  will  operate  satisfactorily  or  will 
fulfill  its  intended  purpose  under  specified  conditions. 

Once  a TSO  has  been  authorized  for  a particular  equipment,  the  manu- 
facturer must  produce  the  equipment  in  accordance  with  his  application, 
conduct  all  required  tests  and  inspections , and  establish  and  maintain  a 
quality-control  system  adequate  to  ensure  that  the  equipment  meets  the 
requirements  of  the  TSO. 

4. 1.1. 3 Summary 

The  commercial  airlines'  process  for  the  procurement  of  avionic  and 
electronic  equipment  has  evolved  over  the  past  35  years.  The  process  in- 
volves the  highly  competitive,  open  forum  participation  of  several  public 
and  private  organizations  in  addition  to  the  user  and  supplier. 

The  airline  procurement  process  is  relatively  simple  when  contrasted 
to  military  processes;  it  enables  the  airlines  to  acquire  highly  reliable, 
state-of-the-art  electronic  systems  offering  excellent  cost  benefits.  It 
is  the  airline's  opinion  that  the  method  provides  them  with  good  value 
for  the  dollars  expended. 
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4.1.2  Operational  Assessment 

As  an  extension  of  the  certification  program  for  new  acquisitions,  the 
FAA  maintains  an  active  role  in  monitoring  the  failure  occurrences  rate  and 
maintenance  practices  of  the  aviation  community.  The  FAA  is  also  concerned 
with  the  field  performance  of  systems  it  acquires  and  operates. 

4 . 1 . 2 . 1 Reliability-Maintenance  Program 

For  each  aircraft  type  certificated, the  FAA  establishes  a maintenance 
and  inspection  program  to  be  followed.  The  operator  must  maintain  appro- 
priate records  to  verify  that  he  has  been  following  the  procedures 
established. 

Some  of  the  air  carriers  believed  that  the  program  was  overly  restric- 
tive and  that  the  standards  did  not  always  track  actual  experience.  As  a 
result,  in  cooperation  with  the  FAA,  they  developed  a reliability  program 
which  provided  the  carriers  more  flexibility  while  continuing  FAA  control.* 
The  essence  of  the  concept  is  that  the  carrier  is  permitted  to  define  its 
own  maintenance  concept,  and  may  change  it  as  the  carrier  sees  fit.  It 
must,  however,  maintain  a data  system  which  shows  at  all  times  that  af- 
fected systems  components  are  within  established  failure  rate  bounds. 

The  maintenance  planning  technique  used  by  the  airline  community,  the 
Airline/Manufacturer  Maintenance  Program  Planning  Document,  is  known  as 
"MSG-2".  It  entails  a systematic  review  of  all  possible  maintenance  tasks 
in  the  context  of  five  basic  questions. 

1.  Is  reduction  in  failure  resistance  detectable  by  routine  flight 
crew  monitoring? 

2.  Is  reduction  in  failure  resistance  detectable  by  in-situ  mainte- 
nance or  unit  test? 

3.  Does  the  failure  mode  have  a direct  adverse  effect  upon  operating 
safety? 

4.  Is  the  function  hidden  from  the  view  of  the  flight  crew? 

5.  Is  there  an  adverse  relationship  between  age  and  reliability? 

Each  of  the  questions  is  asked  in  isolation  from  the  others.  Items 
which  receive  yes  answers  for  Questions  3 and  4 are  candidates  for  some 
type  of  positive  maintenance  action.  Yes  answers  for  Questions  2 and  5 
may  be  candidates  if  the  maintenance  action  is  determined  to  be  practical. 


*FAA  Advisory  Circular  AC  120-17,  "Handbook  for  Maintenance  Control  by 
Reliability  Methods",  December  1964. 


C-7 


i 


The  following  three  types  of  positive  maintenance  actions  are 
recognized : 

• Hard  Time  Tasks  (HT)  - Item  is  removed  from  service  at  specified 
intervals . 

• On-Condition  (OC)  - Periodic  tests  are  made  to  assure  item  is 
meeting  performance  requirement  with  item  being  removed  from 
service  when  found  deficient. 

• Condition  Monitoring  (CM)  - Item  is  removed  from  service  when 
observed  to  be  deficient. 

The  typical  distribution  of  maintenance  actions  among  these  categories 
for  several  selected  aircraft  types  are  shown  in  Table  C-l. 


Table  C- 

2 . DISTRIBUTION  OF  POSITIVE 

MAINTENANCE  ACTIONS 

Aircraft 

Total  Action 

Percent 

HT 

OC 

CM 

DC- 9 

1260 

17 

19 

64 

727 

2078 

17 

25 

58 

707 

3010 

18 

27 

55 

L-1011 

4539 

4 

9 

87 

747 

5908 

6 

22 

72 

In  the  newer  aircraft  designs  (L-1011  and  747)  significant  progress 
has  been  made  reducing  the  hard-time  and  on-condition  maintenance  require- 
ments. This  has  been  accomplished  through  better  fault  monitoring  and 
redundancy. 

For  a new  aircraft,  the  several  airlines  buying  the  aircraft  form  a 
team  to  perform  the  MSG-2  analysis  jointly.  The  aircraft  manufacturer 
supplies  the  initial  task  data  the  committee  uses  for  its  analysis.  The 
FAA  is  invited  to  participate  since  the  completed  analysis  is  submitted 
to  the  administration  for  review. 

Some  airlines  do  not  apply  the  MSG-2  analysis  to  structural  items  al- 
though the  procedure  is  designed  to  cover  them.  They  believe  there  is 
insufficient  information  regarding  those  items  to  justify  a maintenance 
program. 
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A key  part  of  the  analysis  is  setting  the  intervals  between  mainte- 
nance actions  for  the  HT  and  OC  items.  The  airlines  typically  start  with 
intervals  that  have  been  used  for  similar  items  in  existing  aircraft.  As 
they  gain  experience,  the  intervals  are  adjusted  upward  or  downward  as  deemed 
appropriate. 

The  DoD  has  widely  used  the  MSG-2  concept  as  a tool  for  planning 
maintenance  on  military  aircraft. 

4. 1.2. 2 Mechanical  Reliability  Report 

As  a further  control,  the  FAA  requires  Mechanical  Reliability  Reports 
(MRRs)  from  air  carriers. 

All  domestic  scheduled  air  carriers  are  required  to  use  and  implement 
the  MRR  system  by  CFR  Title  14,  Rule  No.  121.703.  Each  carrier  is  required 
to  report  the  occurrence  or  detection  of  each  failure,  malfunction,  or 
defect  related  to  sixteen  specific  conditions  listed  in  Table  C-2. 

Other  conditions  may  be  reported  by  the  carriers. 

The  main  characteristics  of  the  MRR  system  follow: 

• The  carrier  submits  an  MRR  to  the  FAA  Principal  Inspector  within 
24  hours  after  the  end  of  the  previous  24-hour  period  in  which  an 
event  occurred.  The  carrier  uses  its  own  reporting  form;  one  FAA 
Principal  Inspector  is  assigned  to  each  carrier. 

• The  Principal  Inspector  transcribes  the  data  into  the  FAA  MRR 
system  and  forwards  it  to  the  Maintenance  Analysis  Center  (MAC) 
within  24  hours. 

• Every  day,  the  MAC  Data  Bank  prepares  a "Flight  Standards  Service 
Difficulty  Report"  (FSSDR) , which  tabulates  total  occurrences 
entered  into  the  data  bank  on  a specific  date. 

• After  the  carrier's  report  has  been  reviewed  for  completeness,  it 
is  entered  and  stored  as  an  open  or  closed  MRR  in  the  MAC  Data 
Bank.  The  FSSDRs  are  mailed  from  MAC  within  two  to  three  days 
after  receipt  of  the  MRRs  from  the  Principal  Inspector. 

. The  FSSDRs  are  automatically  mailed  to  all  carriers,  prime  air- 
craft manufacturers,  submanufacturers,  and  other  subscribers  to 
the  service. 

• Critical  occurrences  are  identified  on  the  FSSDRs  by  an  extra  heavy 
black  line  border. 

• The  status  of  all  open  items  is  reported  periodically  by  MAC. 

• The  carrier  is  required  to  submit  a follow-up  report  on  open  items 
to  the  Principal  Inspector  following  the  analysis  of  the  problem  or 
occurrence.  There  may  be  one  or  more  follow-up  reports  before  the 
occurrence  is  closed. 


C-9 


T 


Table  C-2.  EVENTS  REQUIRING  AN  FAA  MECHANICAL  RELIABILITY  REPORT 

Fires  during  flight  and  failure  of  the  related  fire-warning  system 
Fires  during  flight  not  protected  by  a related  fire-warning  system 
False  fire  warnings  during  flight 

Engine  exhaust  systems  that  cause  damage  during  flight  to  an  engine, 
adjacent  structure,  equipment,  or  components 

Aircraft  corqponents  that  cause  accumulation  of  circulation  of  smoke, 
vapor,  or  toxic  or  noxious  fumes  in  the  crew  compartment  or  passenger 
cabin  during  flight 

Engine  shutdowns  during  flight  because  of  flameout 

Engine  shutdowns  during  flight  as  a result  of  external  damage  to  the 
engine  or  airplane  structure 

Engine  shutdowns  during  flight  due  to  foreign  object  ingestion  or  icing 
Shutdowns  during  flight  of  more  than  one  engine 

Failure  of  propeller  feathering  system  or  of  the  exility  of  the  system 
to  control  overspeed  during  flight 

A fuel  or  fuel-dumping  system  that  adversely  affects  „uel  flow  or  causes 
hazardous  leakage  during  flight 

Landing  gear  extensions  or  retractions,  or  opening  or  closing  of  landing 
gear  doors  during  flight 

Brake  system  component  failures  that  result  in  loss  of  brake  actuating 
force  when  the  airplane  is  in  motion  on  the  ground 

Aircraft  structures  requiring  major  repair 

Cracks,  permanent  deformation,  and  corrosion  of  aircraft  structures,  if 
more  than  the  maximum  acceptable  to  the  manufacturer  or  the  FAA 

Failure  of  aircraft  components  or  systems  that  result  in  emergency 
actions  during  flight  (except  action  to  shut  down  an  engine) 


• Within  the  FAA  it  is  the  responsibility  of  MAC  to  issue  the  open- 
item  status  report  and  it  is  the  responsibility  of  the  FAA  Principal 
Inspector  to  see  that  the  open  reports  are  closed  out.  However,  the 
carrier  has  the  over-riding  responsibility  to  close  out  each 
occurrence. 

• The  data  bank's  main  uses  are: 

•*  To  advise  the  FAA  of  problems  and  their  current  status 
• • To  advise  carriers  of  occurrences 

••  To  advise  prime  aircraft  manufacturers  of  occurrences 


C-10 


Acquisition  of  systems  within  the  FAA  for  use  in  the  National  Airspace 
System  is  guided  by  several  key  documents.  The  scope  of  these  documents  is 
outlined  in  the  following  paragraphs. 

Electronic  Equipment,  General  Requirements.  FAA-G-2100,  the  electronic 
equipment  general  requirements  specification  (Key  Document  C-l) , outlines 
the  primary  considerations  a vendor  must  address  when  building  equipments. 
The  specification  consists  of  five  parts: 

• Part  1:  Electronic  Equipment,  Basic  Requirements  for  All  Equipments 

• Part  2:  Requirements  for  Equipments  Employing  Electron  Tubes 

• Part  3:  Requirements  for  Equipments  Employing  Semiconductor  Devices 

• Part  4:  Requirements  for  Equipments  Employing  Printed  Wiring 

Techniques 

• Part  5:  Requirements  for  Equipments  Employing  Microelectronic 

Devices 

These  documents  in  turn  cite  and  invoke  selected  sections  of  Department  of 
Defense  Specifications,  including: 

• MIL-STD-454  - Standard  General  Requirements  for  Electronic  Equipment 

• MIL-STD-785  - Reliability  Program 

• MIL-HDBK-217  - Reliability 

• MIL-STD-470  - Maintainability  Program 

By  further  reference,  DoD  reliability  and  maintainability  demonstration 
standards  781  and  471  are  also  invoked  when  necessary.  Other  FAA  standards 
control  other  aspects  of  equipment  procurement,  (e.g.,  quality  control, 
finances) . 

Reliability  and  Maintainability  Policy,  FAA  6000.26.  This  document, 
issued  in  August  1977,  establishes  reliability  and  maintainability  policy 
for  programs  associated  with  acquisition  and  support  of  the  National  Air- 
space System.  Major  objectives  are  to  (1)  establish  R&M  program  require- 
ments for  system  acquisition  and  support  process,  (2)  require  and  obtain 
deliveries  of  systems  with  specified  R&M,  and  (3)  assure  that  operational 
systems  are  performing  in  accordance  with  expectations  and  potential  R&M 
improvements  are  identified. 

R&M  System  Engineering  Program  (Key  Document  C-2  ) . A comprehensive 
reliability  and  maintainability  program  plan,  AAF  200,  was  implemented 
within  the  FAA  in  August  1976.  The  plan  provides  for  the  development  of  a 
series  of  planning  and  implementation  documents  and  the  performance  of  R&M 
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engineering  tasks  for  new  procurements  and  for  fielded  systems.  The  scope 
of  these  efforts  are  shown  in  the  following  outlines: 

Planning  Documents 

(A)  R&M  System  Engineering  Program  Plan 

• Establishes  overall  R&M  goals  (e.g.,  zero  maintenance  growth) 

• Defines  levels  and  procurement  types 

• Defines  tasks  for  new  procurement 

• Defines  tasks  for  field  improvements 

• Defines  AAF  documentation  requirements 

(B)  Optimum  R&M  Level  Determination  Guidebook  for  FAA  Hardware  Sys- 
tems Provides  Criteria  and  Procedures  to  Optimize  MTBF  and  MTTR 
Specifications 

(C)  Guidebook  for  FAA  Systems  Availability 

• Defines  availability  improvement  techniques 
••  Reliability 

• • • Components 
• • • Derating 
•••  Burn-in 
• • Maintainability 
• • • Remoting 

• • • Condition  monitoring  and  fault  isolation 
• • • Modularity 
•••  Logistics 

••  Reliability,  Availability,  and  Maintainability  (RAM)  Program 
• • • Zero  maintenance  growth 
•••  Life-cycle  cost 

(D)  Guidebook  for  Contractor  Development  Programs 

• Defines  program  provisions  and  techniques 

• Defines  deliverables  (documentation) 

• Defines  planning  and  control  requirements 
• • Component  engineering 

• • Design  reviews 
• • Data  collection 
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••  Prediction  (R&M/ 

••  Failure  mode  effects  criticality  analysis  (FMECA) 
••  Part  control 
••  Test  and  evaluation 
• • Scheduling 
• • Cost  control 

(E)  Guidebook  for  Monitoring  Contractor  Programs 

• Establishes  adequacy  of  provisions  and  techniques 

• Establishes  timeliness  of  submittals 

• Establishes  monitoring  of: 

••  Preliminary  and  critical  design  reviews 

■*  • On-site  reviews 

••  Planning  and  control  provisions 

••  R&M  systems  engineering  provisions 

• • Component  engineering  provisions 

• • Test  and  evaluation  provisions 

(F)  Maintainability  Engineering  Provides  Guidelines  for: 

• Remoting 

• Condition  monitoring  and  fault  isolation  (CM  and  FI) 

• Modularity 

• Automation 

••  Built-in- test , fault  isolation  test  (BIT/FIT) 

• • Diagnostics 

(G)  R Systems  Engineering 

• Provides  prediction  procedures 

• Establishes  failure  mode  analysis  techniques 

• Defines  design  review  methods 

• Provides  failure  recurrence  control  procedures 

• Establishes  maintenance  analysis  techniques 

• Establishes  procedures  for  use  of  R&M  reference  data 

• Provides  redundancy  design  approaches 
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(H)  Component  Engineering  Procedural  Manual 

• Provides  part  technique  selection 

• Provides  part  specification  procedure 

• Establishes  part  control  methods 

• Defines  nonstandard  part  approval  procedures 

• Establishes  spare  provisioning  procedures 

(I)  R&M  Testing  Procedural  Manual  Provides  Demonstration  Procedures 
Covering : 

• Full  (statistical)  demonstrations 

• Limited  (Bayesian)  demonstrations 

• Growth  test  procedures 

• Acceptance  test  procedures 

• Forced  defect  test  procedures 

(J)  Data  Collection  and  Reduction  Procedures 

• Establishes  field  data  collection  methods 

• Establishes  part  failure  mode  and  rate  data  production 
techniques 

• Establishes  production  reject  and  degradation  data  collection 
methods 

• Establishes  factory  and  field  cost  data  collection  procedure 

• Develops  formats  for  field  improvement  recommendations 

• Establishes  requirements  of  R&M  and  LCC  cost  memory  bank 

(K)  Failure  Analysis  Procedures 

' Provides  failure  recurrence  control  procedures 
• • R&M  growth  and  demonstration 
• • Production  acceptance  testing 
••  Operation  (field)  trend  reporting 

• Provides  failure  analysis  procedures 
• • Equipment 

••  Devices  (microcircuits,  high  power  tubes,  etc.) 

(L)  Training  Procedural  Manual 

• Provides  training  procedures 

• Provides  procedures  for  preparation  and  use  of  training  aids 
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R&M  Engineering  Tasks  (New  Procurements) 

(A)  FAA  Activities  Prior  to  Contractor  Performance 

• System  R&M  Integration  Studies 

• • Establish  R&M  requirements  for  COO  limits 

••  Identify  cost-effective  procurement  type 

••  Perform  trade-off  analysis  among  reliability,  maintain- 
ability, and  cost 

• • Identify  critical  components 

• Procurement  Package  Review 

••  Review  specified  R&M  values 
• • Review  specified  R&M  program  provisions 
• • Review  compliance  requirements 
• • Review  documentation  requirements 

• * Review  part  control  and  subcontractor  control  provisions 

• Liaison  with  Contractors 

• • Assess  and  monitor  contractor  R&M  efforts 
• • Coordinate  internal  FAA  R&M  efforts 

(B)  FA’'  Activities  During  Contractor  Performance 

• R&M  Assessment  and  Analysis 

••  Perform  independent  R&M  analyses  (special  studies) 

• • Assess  reliability  of  software 

• * Perform  human  engineering  analysis 

• • Identify  production  R&M  degradation  factors 
••  Conduct  design  reviews 
• • Perform  maintenance  analyses 

• * Identify  R&M  and  cost-effective  improvements 

• Part  Control  Activity 

••  Prepare  and  maintain  project  preferred  parts  list 

• • Manage  part  approval  control  function 

••  Define  qualification  and  data  requirements  for  parts 

• R&M  Compliance  Activity 

••  Review  and  evaluate  R&M  test  plans  and  procedures 
• • Retain  R&M  and  cost  memory  bank  data 
• • Coordinate  hardware  failure  analysis 
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RSM  Systems  Engineering  Tasks  (Fielded  Systems) 

• R&M  Assessment  of  Fielded  Systems 

• * Determine  R&M  Field  degradation  factors 

••  Compare  actual  R&M  performance  with  expected  values 

• • Assess  local  operating  procedures 

• R&M  Improvement 

• * Select  items  for  R&M  improvement  studies 

••  Formulate  cost-effective  R&M  improvement  recommendations 
• • Evaluate  R&M  improvements  for  cost-effectiveness 
••  Monitor  effectiveness  of  changes  incorporated 

• Failure  Analysis 

••  Select  critical  components  for  failure  analysis 

• • Coordinate  hardware  failure  analyses 

••  Prepare  summary  failure  reports  and  failure  alerts 

• Data  Collection 

• * Collect  R&M  and  cost  data 

• • Coordinate  data  collection  with  failure  analysis  and  other 
efforts 

••  Identify  and  list  high-failure  and  high-downtime  items 
•*  Reduce  data,  prepare  failure  mode,  and  rate  reports 
• • Prepare  summary  statistical  reports 

System  Acquisition  Publications 

• System  Acquisition  Management  - FAA  1810.1  - This  document  states 
a system  management  policy  providing  for  an  explicit  evaluation  of 
mission  needs  and  program  objectives  to  assure  that  the  process 
for  acquiring  systems  is  efficiently  and  effectively  accomplished. 
Contained  within  this  document  are  a management  framework  and 
procedures  to  be  used  in  the  acquisition  of  major  systems.  Major 
objectives  to  be  sought  in  system  acquisition  are  stated: 

••  Each  system  acquisition  is  directed  toward  fulfillment  of  a 
mission  need. 

••  The  level  of  performance,  maintainability,  and  reliability 
is  in  balance  with  the  allocation  of  resources. 

••  Appropriate  trade-offs  are  considered  among  life-cycle  costs, 
time  schedules,  and  performance  characteristics. 

••  Strong  management  checks  and  balances  are  provided. 

• • An  acquisition  strategy  including  logistics  support  for  each 
system  is  planned  and  refined  throughout  the  acquisition  cycle. 
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••  A capability  is  maintained  to:  (1)  estimate  life-cycle  costs; 
(2)  predict,  review,  assess,  and  monitor  costs  for  system  de- 
velopment, engineering,  design,  demonstration,  test,  produc- 
tion, operation,  and  support;  (3)  assess  cost  schedules  and 
performance  experience  against  predictions,  and  provide  such 
assessments  for  consideration  by  the  Administrator  or  other 
top  management  officials  at  key  decision  points;  and  (4)  make 
new  determinations  where  significant  cost,  schedule,  or 
performance  variances  occur. 

• Major  System  Acquisition  Review  and  Approval  - DOT  4200  - This 
document  was  issued  in  1977  and  was  developed  in  response  to  OMB 
Circular  A109  and  describes  the  approach  to  be  used  in  acquiring 
major  systems.  Major  life-cycle  phases  are  outlined  in  Table  C-3. 


Table  C-3.  KEY  DECISION  POINTS  IN  SYSTEM  LIFE  CYCLES 

Key 

Decision 

Points 

Life-Cycle  Phases 

Starting 

point 

Mission  Needs,  Identification,  and  Designation  of  Major 

Systems 

• Identify  mission  needs 

• Develop  program  to  satisfy  needs 

• Prepare  initial  acquisition  paper  and  program/ 
project  plan 

No.  1 

Research  Phase  and  Exploratory  Development  Phase 

• Potential  system  design  concept  studies 

• Preliminary  research 

• Exploratory  subsystem  development 

• Update  acquisition  paper 

No.  2 

Advanced  Development  Phase  and  Prototype  Development 

Phase 

• Design 

• Fabrication 

• Test 

• Evaluation 

• Update  acquisition  paper 

No.  3 

Preliminary  Operational  Deployment  or  Demonstration  Phase 

• Full-scale  (production)  development 

• Independent  tests  of  system  performance 

• Demonstration  in  expected  operational  environment 

• Limited  production 

• Update  acquisition  paper 

No.  4 

Operational  Phase  — full  production 

4.2.2  FAA  System  Assessment 


The  FAA  has  developed  the  Maintenance  Automated  Reporting  System 
(MARS) , a computer  program  for  the  collection  and  dissemination  of  mainte- 
nance information  regarding  FAA  facilities  in  the  National  Airspace  System. 
MARS  would  use  the  computers  already  in  place  at  the  20  NAS  En  Route  Cen- 
ters, which  would  in  effect  become  system  collection  and  dissemination 
points.  The  system  keeps  track  of  outages  and  provides  individual  system 
histories  to  which  technicians  can  refer  when  correcting  failures  or  working 
up  schedules  for  preventive  maintenance. 

A further  part  of  the  AAF  200  R&M  plan  (part  of  Key  Document  C-2;  is 
the  development  of  an  air  R&M  data  bank  and  collection  system.  Data  to  be 
obtained  for  the  system  will  include  equipment  and  classification  informa- 
tion as  well  as  equipment  experience  data.  When  the  data  system  is  fully 
developed  and  implemented,  it  will  provide  output  information  such  as 
reliability  and  maintainability  parameters  and  cost  data  to  support  the 
R&M  system  engineering  program  (both  documentation  and  engineering  tasks) . 


4. 3 Lead  Responsibilities  and  Interface 


The  FAA  assumes  the  lead  role  in  assuring  safety  in  the  airways  sys- 
tem; that,  in  turn,  provides  good  system  reliability  as  a direct  benefit. 
To  fulfill  this  role,  the  FAA  has  interfaces  with  the  air  carriers,  equip- 
ment suppliers,  and  other  government  agencies  such  as  the  NT SB  and  the  CAB 
These  relationships  have  developed  over  a period  of  years  paralleling  the 
growth  of  aviation. 


5.  COST  AND  EFFECTIVENESS 
5.1  Program  Cost 

The  specific  amount  spent  by  FAA  on  performance  assurance  is  not  known, 
but  a review  of  some  of  the  budget  requests  for  FY  1978  could  provide  some 
insight. 

FAA's  total  budget  request  for  1978  was  $1,819,750,000.  Of  this, 
$203,389,000  was  earmarked  for  administration  of  flight  standards.  This 
includes  all  of  the  certification  efforts,  safety  programs,  and  inspection 
of  flight  facilities. 

The  1978  equipment  acquisition  budget  to  supplement  the  National  Air- 
ways System  was  $212,600,000.  It  is  estimated  that  $6,990,000  of  this 
amount  will  be  spent  on  development  and  test  and  $469,646,000  on  mainte- 
nance of  the  systems. 


5.2  Program  Effectiveness 


FAA  has  been  faced  with  severe  technological  growth  problems  as  it 
has  taken  steps  to  automate  the  National  Airways  System  to  improve  its 
productivity.  As  noted,  its  performance  assurance  for  these  systems  is 
similar  in  many  respects  to  the  DoD  concepts  and  its  results  are  similar. 

Systems  acquired  by  the  air  carriers  subject  to  FAA  certification 
processes  use  a different  approach,  which  has  been  shown  to  be  cost- 
effective  in  producing  reliable  and  safe  systems.  This  approach  places 
most  emphasis  on  testing  (certification)  of  the  final  product  design  and 
the  product's  being  capable  of  performing  its  required  function  reliably. 
The  air  carriers  through  the  development  of  system  standards  and  their 
insistence  on  unity  commitments  aid  further  good  reliability  achievement. 
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PERFORMANCE  ASSURANCE  PRACTICES  OF  THE 
NATIONAL  TRANSPORTATION  SAFETY  BOARD 


1.  HISTORICAL  DEVELOPMENT 

The  National  Transportation  Safety  Board  (NTSB)  was  created  by  the 
Department  of  Transportation  Act  of  1966,  making  the  Board  a part  of  the 
Department  of  Transportation.  The  Independent  Safety  Board  Act  of  1974 
established  the  Safety  Board  as  an  entirely  independent  Federal  agency, 
and  broadened  its  responsibilities  in  the  investigation  and  prevention  of 
transportation  accidents.  The  1974  Act  directed  the  Board  to  report  to  the 
Congress  on  July  1 of  each  year. 


2.  SCOPE  OF  CURRENT  PRACTICES 
2.1  NTSB  Charter 

Under  its  current  charter  NTSB  is  charged  with: 

• Investigating  certain  aviation,  highway,  railroad,  pipeline,  and 
marine  accidents 

• Reporting  publicly  on  the  facts,  conditions,  and  circumstances,  and 
the  cause  or  probable  cause  of  such  accidents 

• Issuing  periodic  reports  to  the  Congress  and  to  Federal,  state, 
and  local  transportation  safety  agencies  and  others  recommending 
measures  to  reduce  the  likelihood  of  transportation  accidents 

• Initiating  and  conducting  special  transportation  safety  studies  and 
investigations 

• Assessing  accident  investigation  methods  and  publishing  periodic 
recommendations  on  investigation  procedures 

• Establishing  requirements  for  reporting  accidents  to  the  Board 

• Evaluating  and  publishing  findings  on  the  transportation  safety 
consciousness  and  accident  prevention  efficacy  of  other  government 
agencies 

• Evaluating  the  adequacy  of  hazardous  materials  transportation  safe- 
guards and  procedures 
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• Reviewing  on  appeal  the  suspension,  amendment,  modification,  revoca 
tion,  or  denial  of  certain  operating  certificates,  documents,  or 
licenses  issued  by  the  Federal  Aviation  Administrator  and  by  the 
Commandant  of  the  Coast  Guard 

2.2  Key  Documents 

The  responsibility  and  authority  of  the  National  Transportation  Safety 
Board  are  derived  from: 

• The  Transportation  Safety  Act  of  1974,  Title  III:  "Independent 
Safety  Board  Act  of  1974"  (88  Stat.  2156,49  U.S.C.  1901) 

• The  Federal  Aviation  Act  of  1958,  August  23,  1958,  as  amended  (72 
Stat.  731,49  U.S.C.  1301) 

• The  Federal  Railroad  Safety  Act  of  1970,  October  16,  1970  (84  Stat. 
791,45  U.S.C.  421) 

Regulations  of  the  National  Transportation  Safety  Board  are  published 
in  the  Federal  Register  and  codified  in  the  Code  of  Federal  Regulations, 
Chapter  VIII,  Title  49  - Transportation. 

2 . 3 Organizational  Structure 

NTSB  is  structured  into  the  following  four  major  bureaus. 

2.3.1  Bureau  of  Accident  Investigation 

The  Bureau  of  Accident  Investigation  is  responsible  for  all  accident 
investigations  in  the  five  modes  of  transportation:  aviation,  marine,  rail 
road,  highway,  and  pipeline.  To  aid  the  work  of  the  Bureau,  there  are  12 
field  offices. 

2.3.2  Bureau  of  Technology 

The  Bureau  of  Technology  serves  as  the  Safety  Board's  reservoir  of 
technical  expertise.  Specialists  from  the  Bureau  provide  support  for  both 
the  Board's  investigative  and  accident  prevention  activities.  For  example, 
specialists  in  the  Division  of  Human,  Vehicle,  and  Operational  Factors  as 
well  as  the  Hazardous  Materials  Division  contribute  to  accident  investiga- 
tions and  take  part  in  public  hearings  as  members  of  technical  panels.  In 
aiding  foreign  accident  investigations,  this  service  includes  using  the 
Bureau's  laboratory  to  examine  vehicle  parts  or  aircraft  flight  data  and 
voice  recorders  recovered  for  foreign  investigatory  bodies. 

2.3.3  Bureau  of  Plans  and  Programs 

The  Bureau  of  Plans  and  Programs  was  created  to  develop  and  manage  the 
Safety  Board's  accident  prevention  and  safety  promotion  programs.  These 
programs  include  planning  and  conducting  special  studies.  The  Bureau  also 
is  responsible  for  determining  the  Safety  Board's  personnel  training 


requirements,  planning  safety  program  evaluations,  and  management  reviews 
of  safety  activities.  Finally,  the  Bureau  is  responsible  for  both  proposing 
and  advocating  changes  in  Safety  Board  policy  in  the  area  of  transporation. 

3.2.4  Bureau  of  Administration 

The  Bureau  of  Administration  was  established  to  provide  unified  direc- 
tion and  management  of  the  Safety  Board's  administrative  programs.  These 
programs  include  financial  and  personnel  management,  management  analysis, 
and  operations  and  facilities. 


3.  APPLICABILITY  OF  SELECTED  PROGRAMS 

The  current  charter  of  the  MTSB  permits  it  to  have  an  impact  on  the 
electric  power  industry  under  its  responsibilities  for  the  transportation 
of  hazardous  materials.  Its  function  as  an  independent  investigator  of 
accidents  within  the  transportation  industry  is  being  met  by  NR C and  local 
government  agencies.  However,  it  is  considered  of  interest  to  view  the 
approaches  used  by  NTSB  to  gain  insight  toward  reliability  enhancement 
insofar  as  accident  investigation  techniques  can  be  converted  into  power 
outage  investigation  techniques. 


4.  DESCRIPTION  OF  APPLICABLE  PRACTICES 
4. 1 Key  Program  Elements 

Major  efforts  of  the  NTSB  are  directed  toward: 

• Accident  Investigation 

• Safety  improvement  and  recommendations  program 
Highlights  of  the  activities  in  each  area  are  outlined. 

4.1.1  Accident  Investigation 

During  1976  the  NTSB  investigated  846  aircraft  accidents  and  reviewed 
the  3448  accidents  investigated  by  FAA.  Additionally,  the  Board  investigated 
9 highway,  12  railroad,  5 pipeline,  and  2 marine  accidents. 

The  scope  of  an  MTSB  accident  investigation  is  suggested  by  the  follow- 
ing outline  of  a typical  aircraft  accident  report. 

• Synopsis 

• Investigation 

••  History  of  flight 
••  Injuries  to  persons 
••  Damage  to  aircraft 
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• • Other  damage 

• • Crew  information 

••  Aircraft  information 

• • Meteorological  information 

• • Aids  to  navigation 

• * Communications 

••  Aerodrome  and  ground  facilities 
• • Flight  recorders 
• • Wreckage 

• • Medical  and  pathological  information 

• * Fire 

• • Survival  aspects 
• • Tests  and  records 
• • Other  information 

• Analysis  and  Conclusions 
• • Analysis 

••  Conclusions 

• Recommendations 

• Appendix  A - Investigation  and  Hearing 

• Appendix  B - Crew  Information 

• Appendix  C - Aircraft  Information 

• Appendix  D - Approach  Data 

• Appendix  E - Tower  Transcript 

• Appendix  F - Flight  Track 

• Appendix  G - Safety  Recommendation 

• Appendix  H - Specialist  Report 

4.1.2  Safety  Improvement  and  Recommendations  Programs 

During  1976  major  NTSB  activities  for  inproving  safety  conditions  are 
highlighted  as  follows: 

• Aviation  Safety 

• • Reducing  approach  and  landing  accidents 
••  Improved  accident  survivability 
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Highway  Safety 


1 

••  Improved  traffic  barrier  crashworthiness 
••  Reduced  constructions  zone  hazard 
••  Coordination  of  vehicle  bumper  standards 
••  Safety  belts  for  intercity  buses 
• • National  driver  register 

• Marine  Safety 

• • Lighting  of  barges 

••  Structural  integrity  of  tank  ships 

• • Aids  to  marine  investigations 

• Pipeline  Safety 

••  Specialized  regulations  of  highly  volatile  liquids 
• • Maintaining  of  pipeline  safety  regulations 
••  Protection  of  pipeline  against  construction  damage 

• Railroad  Safety 

• • Safety  standards  for  rail  rapid  transit 
••  Collision  avoidance 
• • Aids  to  accident  investigation 
• • Hazardous  materials 

The  need  for  such  projects  is  determined  by  the  results  of  accident 
investigations  conducted.  As  a result  of  these  studies,  NTSB  makes,  as 
appropriate,  specific  recommendations  to  Congress  for  new  laws  governing 
safety. 

4. 2 Lead  Responsiblities  and  Interfaces 

NTSB  has  the  lead  responsibility  for  accident  investigation  as  chartered 
by  the  enabling  legislation.  However, in  the  discharge  of  this  responsibility 
it  interfaces  with  several  other  government  agencies  including: 

• FAA 

• The  Bureau  of  Motor  Carrier  Safety 

• National  Highway  Traffic  Safety  Administration 

• The  Office  of  Pipeline  Safety  Operations 

• U.S.  Coast  Guard 

• Federal  Railroad  Administration 

These  agencies  may  support  NTSB  in  the  conduct  of  an  investigation  or  may 
conduct  the  total  investigation  with  the  results  being  reviewed  by  NTSB. 
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Industry  participates  as  part  of  the  NTSB  investigation  activity  and  in 
studies  leading  to  possible  changes  in  safety  rules  as  they  apply  to  the 
transportation  field. 


5.  COST  AND  EFFECTIVENESS 
5.1  Cost 


The  cost  of  running  the  NTSB  in  FY  1977  was  $13,800,000  and  for  FY  1978 
is  forecast  to  be  $14,710,000.  Of  this,  approximately  one-half  is  spent  on 
accident  investigation. 

5. 2 Program  Effectiveness 

The  results  of  the  accident  investigation  and  the  other  studies  conducted 
by  NTSB  has  led  to  the  identification  of  technological,  procedural , environ- 
mental, or  operational  causes.  This  identification  has  led  in  many  cases  to 
ameliorative  actions  which  precluded  further  occurrences.  Considering  the 
magnitude  of  the  potential  casualty  loss  of  a large  aircraft,  the  cost  of 
accomplishing  the  investigation  and  analysis  seems  justified. 
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PERFORMANCE  ASSURANCE  PRACTICES  OF  THE 
URBAN  MASS  TRANSIT  ADMINISTRATION  (UMTA) 


1.  HISTORICAL  DEVELOPMENT 

Since  1964,  UMTA  has  provided  capital  assistance,  technical  assistance, 
and  operating  subsidies  to  communities  for  improving  existing  mass  transpor- 
tation systems  and  developing  new  transit  systems.  UMTA  is  supporting  tran- 
sit construction  in  Atlanta,  Baltimore,  Philadelphia,  Buffalo,  Detroit, 
Miami,  Houston,  Cleveland,  Los  Angeles,  and  St.  Paul. 

The  principal  role  of  UMTA  has  been  to  act  as  a funnel  through  which 
Federal  money  flows  back  to  the  cities,  states,  and  certain  private  compa- 
nies. Increasingly,  UMTA  is  interested  in  improving  the  effectiveness  of 
both  Federal  investment  and  investment  by  local  communities.  Therefore, 
over  the  last  two  to  three  years,  UMTA  has  broadened  its  previous  interest 
in  safety  as  the  primary  measure  of  system  performance  to  include  life- 
cycle  cost  and  system  reliability,  maintainability,  and  availability  as 
important  measures  of  system  performance. 

The  increasing  emphasis  on  improving  performance  assurance  was  prompted 
by  the  very  low,  initial,  service  availability*  of  advanced  transit  systems 
(e.g.,  the  San  Francisco  Bay  Area  Rapid  Transit  System  - BART)  and  special- 
ized demonstration  systems  (e.g.,  the  Morgantown  People  Mover)  that  had 
received  UMTA  funds.  UMTA  was  also  criticized  by  the  Government  Accounting 
Office  (GAO)  in  1976  for  its  inability  to  assure  the  reliability  of  rail 
cars  purchased  by  the  New  York  Transit  Authority  and  communications  equip- 
ment purchased  by  the  Chicago  Transit  Authority.  UMTA  has  responded  to 
these  criticisms  by  developing  a performance  assurance  program  as  described 
in  the  remainder  of  this  section. 


*UMTA  employs  the  concept  of  "service  availability"  in  such  a way  that  the 
system  is  declared  unavailable  if  any  passenger  cannot  complete  his  trip  in 
a prescribed  nominal  time. 
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2.  SCOPE  OF  CURRENT  PERFORMANCE  ASSURANCE  ACTIVITIES  AT  UMTA* 

Although  UMTA  is  beginning  to  recognize  the  need  for  a more  cohesive 
and  centralized  approach  to  performance  assurance,  there  is  currently  only 
one  focal  point  for  performance  assurance  activities  within  the  agency  — 
the  Mass  Transit  Safety  and  System  Assurance  Program  (MTSSA) . 

As  the  title  of  the  program  suggests,  the  emphasis  within  the  Mass 
Transit  Safety  and  System  Assurance  Program  has  been  on  safety.  However, 
in  FY  1976  the  activity  was  reassigned  to  the  Office  of  Technology  Develop- 
ment and  Deployment.  At  the  same  time,  the  emphasis  was  shifted  to  allow 
the  optimization  of  safety  and  security  in  consideration  of  other  system 
values,  thereby  suggesting  a growing  role  for  UMTA  in  helping  to  improve 
system  availability,  system  dependability,  equipment  maintainability  and 
reliability,  and  life-cycle  cost. 

The  management  of  the  MTSSA  program  is  oriented  to  assist  in  and  con- 
tribute to  the  quality  of  local  and  regional  decision-making  processes 
applied  to  the  development  of  new  transit  systems , and  the  improvement  or 
expansion  of  existing  systems,  based  on  three  considerations: 

1.  Recommend  rather  than  require. 

2.  Recognize  that  ultimate  accountability  for  system  operational 
viability  lies  with  local  or  regional  decision  makers  accountable 
to  the  public  for  system  acquisition  and  operation. 

3.  Avoid  preemption  of  local  accountability  for  decision. 

In  applying  the  MTSSA  program,  UMTA  employs  a functional  approach,  an 
organizational  approach,  and  a management  approach. 

The  functional  approach  follows  these  guidelines:  safety  of  the  system 
is  not  paramount;  safety  and  other  contributors  to  the  operational  viability 
of  a system  are  interdependent  (competitive  and  contributing) ; the  highest 
practicable  level  of  safety  is  not  achievable  without  concurrent  considera- 
tion of  other  contributors  (informed  decisions);  operational  viability  is  a 
system  life-cycle  consideration;  the  focus  is  on  operational  needs;  and  it 
is  better  to  preempt  than  to  react. 


*UMTA  policy  with  respect  to  performance  assurance  and  the  scope  of  UMTA 
performance  activities  are  in  a state  of  transition.  The  summary  offered 
here  is  our  own  interpretation  of  the  changing  scene  and  may  not  reflect 
all  viewpoints  or  adequately  represent  the  result  of  recent  management 
deliberations. 
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The  organizational  approach  includes  the  involvement  of  the  Transporta- 
tion Systems  Center  (TSC)  , the  industry  associations, and  the  Transportation 
Safety  Institute.  Explicit  guidelines  are:  UMTA's  role  is  to  be  limited: 
there  is  a need  for  department  level  participation;  there  is  a need  to  main- 
tain strong  communications  and  interactions  with  experienced  transit  man- 
agers; and  there  is  a need  for  industry  and  government  education  and 
instruction. 

The  management  approach  is  defined  as  a sequential  closed-loop  iterative 
process  undertaken  to:  endorse  and  commit  to  transit;  stimulate  and  assist 
transit;  learn  and  assimilate  from  transit;  and  evaluate,  recommend,  and 
negotiate  with  transit. 

3.  APPLICABILITY  OF  UMTA  METHODS  TO  THE  ELECTRIC  POWER  INDUSTRY 

Local  mass  transit  authorities  rarely  have  the  engineering  expertise 
necessary  to  design  and  manage  the  construction  of  a mass  transit  system. 
Typically, they  delegate  the  management  and  coordination  responsibilities  to 
a prime  contractor,  who  designs  the  system  in  cooperation  with  consultants 
and  manages  subcontractors  to  build  it.  That  practice  is  similar  to  the 
electric  power  industry  retention  of  an  architect-engineer  for  design  and 
to  oversee  procurement  and  construction.  UMTA,  recognizing  that  most  local 
mass  transit  authorities  and  many  prime  contractors  have  no  formal  perform- 
ance assurance  program,  has  provided  guidelines  for  installing  formal 
programs  and  gives  courses  to  acquaint  local  mass  transit  authorities  with 
the  concept  and  implementation  methods.  The  U.S.  Department  of  Energy 
could  also  prepare  program  guidelines  and  give  courses  to  acquaint  electric 
power  utilities  with  performance  assurance  concepts  and  techniques.  The 
U.S.  Department  of  Energy  could  also  sponsor  EPRI , EEI , IEEE, or  another 
industry  association  to  follow  UMTA's  example. 

4.  DESCRIPTION  OF  APPLICABLE  PRACTICES 
4 . 1 Key  Program  Elements 

The  MTSSA  program  is  currently  thrusting  in  two  directions.  One  thrust 
is  to  sponsor  a number  of  education  and  instruction  courses  to  introduce  the 
transit  industry  to  performance  assurance  concepts.  The  other  thrust  is  to 
stimulate  metropolitan  transit  authorities  in  implementing  local  performance 
assurance  programs.  The  Metropolitan  Atlanta  Rapid  Transit  Authority  (MARTA) 
is  the  first  authority  to  make  a commitment  to  undertake  formalized  research, 
review,  and  a systematic  approach  to  the  integration  and  implementation  of 
the  MTSSP  at  the  local  level. 
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4.1.1  Education  and  Instruction 


The  following  instruction  courses  have  been  developed  and  implemented 
or  are  scheduled  for  future  development  and  implementation  (in  cooperation 
with  DOT'S  Transportation  Safety  Institute  in  Oklahoma  City).* 

A.  Introduction  to  Mass  Transit  Safety  and  System  Assurance  - 

A five-day  course  implemented  in  September  1976,  scheduled  to 
be  given  quarterly. 

B.  Quality  Assurance  - A five-day  course,  implemented  in  November 
1976,  scheduled  to  be  given  quarterly. 

C.  System  Safety  - A five-day  course,  implemented  in  January  1977, 
scheduled  to  be  given  quarterly. 

D.  System  Security  - A five-day  course  currently  under  development, 
to  be  implemented  in  FY  1978. 

E.  Reliability,  Maintainability,  Availability,  Dependability  (RMAD)  - 
A five-day  course  currently  under  development  to  be  implemented 

in  November  1977. 

F.  Human  Factors  - A five-day  course  to  be  developed  and  implemented 
in  FY  1978. 

4.1.2  MARTA  Safety  and  System  Assurance  (Performance  Assurance) 
Project  Objectives 

The  objectives  of  the  MARTA  performance  assurance  project  are  summa- 
rized in  Figures  E-l  and  E-2.** 

4.1.3  MARTA  Reliability  Program  Plan 

The  MARTA  Reliability  Program  Plan,  prepared  by  Parsons  and  Brinker- 
hoff,  Tudor  Engineering,  and  Bechtel  (PBTB) , the  general  engineering  con- 
sultants, is  an  excellent  example  of  a complete  performance  assurance 
program  (Key  Document  E-l  ) . An  outline  of  the  planned  reliability  program 
taken  from  Key  Document  E-l  follows: 

A.  Management 

• Delegation  of  authority  - PBTB  to  assist  in  the  development  and 
the  implementation  of  the  reliability  program  and  perform  all 
reliability  analysis  and  related  technical  studies. 

• Organizational  responsibility  - Responsibility  and  line  of 
authority  defined  explicitly  by  title  and  name. 


*This  information  was  taken  from  a widely  distributed  letter  from  George  J. 
Pastor,  Associate  Administrator  for  Technology  Development  and  Deployment, 
UMTA,  July  1977  (project  DC-06-0139). 

**W.  E.  Gooden  and  A.  M.  Lock,  "Safety  and  System  Assurance  Resources 
Applied  to  the  Design  and  Development  of  a Rail  Rapid  Transit  System", 
Third  International  System  Safety  Conference,  Washington,  D.C. , 17-21 
October  1977. 


System  Safety 
Program  Plan 


Reliability  Program  Plan 


Maintainability 
Program  Plan 


Establish  system 
safety  goals  and 
criteria  and  im- 
plement them 
throughout  the 
system. 

Identify  and 
assess  system 
safety  hazards 
as  early  as 
possible  in  the 
design  phase. 

Take  appropri- 
ate actions  to 
eliminate,  min- 
imize, or  con- 
trol the  iden- 
tified critical 
or  catastrophic 
hazards. 

Verify  the  MARTA 
system  as  safe 
for  revenue 
service  prior  to 
opening  date. 


Establish  state-of-the-art 
reliability  requirements, 
specifications,  and 
criteria. 

Ensure  contractor  and  ven- 
dor compliance  with  all 
reliability  requirements, 
specifications,  and 
criteria. 

Verify  the  reliability 
potential  of  the  MARTA 
rail  network  through 
system  and  subsystem 
analyses  and  equipment 
demonstration  testing 
initiated  prior  to 
revenue  service. 

Isolate  and  correct 
potential  reliability 
problems . 

Reverify  the  reliability 
potential  of  the  MARTA 
rail  system  through 
total-system- integration 
testing  initiated  prior 
to  revenue  service. 

Continuously  assess  the 
inherent  reliability  of 
the  MARTA  rail  system  as 
a function  of  failure 
data  collected  during 
testing. 

Demonstrate  achievement  of 
all  reliability  require- 
ments, specifications,  and 
criteria. 


Establish  the  main- 
tenance concept  and 
goals  for  the  MARTA 
system. 

Establish  the  meth- 
ods by  which  the 
maintenance  goals 
will  be  met. 

Incorporate  main- 
tainability concepts 
into  the  design  to 
optimize  maintenance 
with  respect  to  per- 
sonnel, safety,  per- 
sonnel skill  levels, 
reliability,  and 
logistics  support. 

Monitor  maintain- 
ability design 
analyses  and  pre- 
dictions . 

Support  operational 
procedures  relative 
to  maintainability 
concepts  in  areas 
of  design  assembly, 
testing,  installa- 
tion, and  operation. 

Support  the  test  and 
evaluation  program 
for  maintainability 
assistance  in  re- 
pairs, installation, 
and  analyses. 


Figure  E-l.  MARTA  SAFETY  AND  SYSTEM  ASSURANCE  PROGRAM  OBJECTIVES 


Quality  Assurance 
Program  Plan 


Fire  Protection  Discipline 


Security  Discipline 


Assure  that  all 

Provide  MARTA  fire  protection 

Provide  that  security 

work  performed 

and  life  safety  equivalency 

is  conceptually  de- 

for  the  transit 

levels  for: 

signed  and  built  into 

system  is  per- 
formed in  accord- 

• The  prevention  of  fire 

the  system  in  a man- 
ner that  assures  a 

ance  with  the 

• The  protection  of  the 

real  as  well  as  a 

engineering 

general  public,  MARTA 

high  perceptual  level 

requirements . 

employees/ and  fire  de- 

on  the  part  of 

Assure  that  all 
equipment  is 
tested  through- 

partment  personnel  from 
injury  due  to  fire,  smoke, 
explosion,  or  panic 

patrons  and  personnel 
for  the  assurance  of: 

• Patron  and  personnel 

out  development , 

• The  protection  of  MARTA 

safety 

manufacture , and 
installation  to 

structures  and  equipments 
from  damage  due  to  fire 

• System  integrity 

verify  functions 

as  otherwise  provided 

• Patron  assistance 

as  specified. 

through  local  codes  and  as 

Assure  that  un- 
desirable condi- 
tions are  de- 
tected, and 
positive  correc- 
tive action  per- 
formed promptly. 

appropriate  for  the  unique 
aspects  of  a rail  rapid 
transit  system 

Assure  that 
control  over  the 
configuration  is 
maintained  at 
all  times  to  en- 
able timely  cor- 
rection and 
improvements. 

Figure  E-2.  MARTA  SAFETY  AND  SYSTEM  ASSURANCE  PROGRAM  OBJECTIVES 


Reliability  integration  - MARTA  to  coordinate  reliability  efforts 
with  related  disciplines  such  as  maintainability,  safety,  quality 
assurance,  design  and  operations. 

Reliability  standardization  - Emphasis  placed  on  proven  rapid 
transit  design  techniques,  equipment,  and  hardware. 


Equipment  contractor  controls  - MARTA  to  generate  reliability 
specifications,  requirements  related  to  equipment  and  hardware;  all 
major  contractors,  subcontractors,  and  suppliers  to  submit  formal 
reliability  program  plans. 

••  Reliability  design  analysis 

••  Reliability  parts  program  and  controls 

••  Reliability  failure  modes  and  effects  analysis  (FMEA) 

••  Reliability  prediction  analysis 

••  Reliability  data  collection  techniques  and  sources 

••  Reliability  failure  criteria  for  demonstration  and  acceptance 
testing 

••  Reliability  problems 
••  Reliability  progress  reports 

Reliability  program  reviews  - MARTA  to  conduct  periodic  program 
reviews  at  established  reliability  milestones.  The  content  of 
reviews  includes : 

••  Reliability  program  objectives 
••  Current  reliability  requirements,  estimates 
• • Potential  reliablity  problems 
• • Controls  and  procedures 

• • Reliability  parts  applications  and  controls 
• • FMEA 

••  Reliability  trade-offs  and  related  effects  on  other  disciplines 
• • Status  of  reliability  programs 
••  Overall  reliability  impact 
••  Reliability  status  reporting  - monthly 
• • Reliability  consultant  - retained  by  MARTA 
•••  Reliability  analysis 
•••  Reliability  allocations 
•••  Reliability  predictions 
•••  Reliability  trade-off  studies 
• • • FMEA 

• • • Functional  models 
• • • Mathematical  models 

•••  Reliability  training  - MARTA  to  disseminate  requirements, 
specifications,  and  criteria  by  directives,  memoranda, 
procedures,  conference  meetings,  written  and  oral  commu- 
nications , formal  seminars 


B.  Reliability  Design  and  Analysis 

• Design  techniques  - MARTA  to  sign  off  on  design  drawings  and 
technical  data  regarding: 

••  Stress  and  derating  factors 

• • Redundancy 

••  Stress-strength  margins 

• • Nondegraded  performance  capability  at  all  required 
environmental  levels 

••  Criticality,  upgrading 

• • System  and  subsystem  integration 

• • Compatibility  with  diagnostic  test  equipment 

• Stress  analysis  - MARTA  reviews  design  drawings  for  functional 
and  environmental  stress  levels,  considering  real-time  operat- 
ing conditions. 

• • Functional  factors 

• • • Hardware  locations 

• • • Positioning 

• • • Alignments 

•••  Interfacing 

• • Environmental  factors 

• • • Vibration 

• • • Temperature 

•••  H lmidity 

• • • Wind  gusting 

• • Design  reviews  - reliability  specialists  will  support 
engineering  in  reviewing: 

•••  Current  reliability  requirements  and  estimates 

• • • Problem  areas 

• • • Controls  and  procedures 

• • • Parts 

• • • FMEA 

• • • Trade-offs 

•••  Reliability  program  status 

•••  Status  of  previously  approved  design  review  action 
items 
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Reliability  analysis  - system  status  analysis  to  evaluate  impact  of 
performance,  design,  and  operational  objectives  or  the  formulation 
of  reliability  allocations  and  predictions.  Consider: 

••  System  definitions 

••  Functional  flow  diagrams  and  math  models 

*•  Math  model  predictions  in  comparison  with  allocated  requirements 
••  Impact  of  changes  on  reliability  predictions 

Reliability  allocations  - reliability  math  model  to  be  formulated 
early  in  the  program  to  allocate  overall  system  reliability 
requirements  down  to  the  subsystem  and  equipment  levels.  Alloca- 
tion changes  to  be  justified  by  either  contract  or  specification 
revision . 

Reliability  predictions  - MARTA  to  perform  predictions  analysis  of 
system  using  aforementioned  math  model. 

• • Identify  potential  problem  areas 

*•  Provide  a guide  for  additional  inputs  to  the  FMEA 

• * Begin  at  the  piece-part  level 
••  Identify  critical  parts 

•*  Study  system  effectiveness 

• * Provide  historical  data  (by  MARTA) 

Reliability  parts  selection  - preferred  parts  list  supplied  by 
MARTA,  screening  techniques,  predict  reliability  in  consideration 
of  stress  and  environment. 

FMEA  - MARTA  to  analyze  rail  system  to  determine  possible  modes  of 
failure  and  effects  on  revenue  service. 

*•  Identify  critical  failure  areas 

••  Conduct  FMEA  down  to  lowest  replaceable  module  level 
• • Consider 

• • • Module  function 
• * * Means  of  detection 
• • • Corrective  action 
• • • Likelihood  of  failure 
• • • Wear-out 

* • * Performance  degradation 

• • • Environmental  stresses 

•••  Safety  hazards 

•••  Random  catastrophic  failures 

• • • Human  error 
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•••  Documentation  - nomenclature  specified  in  detail 

•••  To  be  used  as  the  baseline  of  a maintainability  engineer- 
ing analysis  (MEA) 


■•i  •••  Input  to  reliability  demonstration  test  plans,  defining 

test  conditions 

• Reliability  demonstration  and  acceptance  testing  - MARTA  to  develop 
and  coordinate  development  of  test  plans  for  critical  systems. 

••  Verify  reliability  design  analysis 

• • Verify  FMEA 

••  Verify  reliability  predictions 
••  Test  parameters  specified  by  MARTA 
• • • Number  of  test  units 
• • • Total  hourly  test  time 
•••  Accept-reject  criteria 
•••  Statistical  confidence  levels 

• Reliability  assessment  - MARTA  to  develop  statistical  assessment 
technique  for  estimating  the  inherent  reliability  of  critical 
systems  based  on  all  research,  development,  and  acceptance  test 
data.  Aimed  at  guiding  corrective  effort  where  reliability 
achievement  falls  short  of  established  requirements. 

• Failure  data  reporting,  analysis,  and  corrective  action  - MARTA  to 
administer  a strictly  controlled  system  for  the  reporting, 
analysis,  correction,  and  data  feedback  on  all  equipment  failures 
detected  during  fabrication,  testing,  and  operation. 

• Reliability  Documentation  - to  be  submitted  by  MARTA 
• • Reliability  program  plan 
••  Reliability  trade-off  studies 
• • Reliability  analysis 

••  Reliability  design  criteria 
••  Reliability  allocations  and  requirements 
••  K..  bility  test  plans  and  reports 

• • F ability  progress  and  status  reports 

• Reliability  Records  and  Files 
• • Traceability 

••  Centralized  information  availability 
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4.2  Lead  Responsibilities  and  Government-Industry  Interface 


The  lead  responsibility  for  assuring  that  the  performance  assurance 
program  is  implemented  lies  with  the  local  mass  transit  authority.  The 
authority  is  motivated  by  UMTA's  requirement  of  the  program  and  reviews  it 
as  a condition  for  allocating  federal  funds  to  the  project.  UMTA  guides 
the  local  authorities  through  these  reviews  and  guides  the  entire  transit 
community  by  means  of  the  UMTA- sponsored  educational  courses. 


5.  COST  AND  EFFECTIVENESS 

Although  detailed  performance  assurance  cost  information  is  not  avail- 
able, it  appears  that  expenditures  for  performance  assurance  by  MARTA  are 
from  0.01  to  0.2  percent  of  system  acquisition  costs. 

The  effectiveness  of  the  program  cannot  be  determined  because  no  mass 
transit  system  has  been  built  and  operated  in  accordance  with  the  UMTA 
guidelines.  However,  systems  built  without  applying  a comprehensive  per- 
formance assurance  program  have  experienced  severe  reliability  and  main- 
tainability problems  and,  hence,  severe  service  availability  problems. 
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APPENDIX  F 


PERFORMANCE  ASSURANCE  AS  PRACTICED  BY 
THE  NUCLEAR  REGULATORY  COMMISSION  (NRC) 


1.  HISTORY  AND  SCOPE  OF  PERFORMANCE  ASSURANCE  ACTIVITIES 

Under  the  Energy  Reorganization  Act  of  1974,  NRC  became  responsible 
for  inplementing  all  regulatory  requirements  of  the  Atomic  Energy  Act  of 
1954,  as  amended.  The  system  of  licensing  and  regulation  devised  to 
carry  out  NRC ' s mission  is  implemented  through  rules  and  regulations  under 
Title  10  of  the  Code  of  Federal  Regulations  (CFR) . An  important  part  of 
this  mission  is  the  consideration  of  applications  to  construct  and  operate 
nuclear  power  plants. 

The  NRC  licensing  process  is  a two-stage  procedure.  The  initial  stage 
consists  of  the  filing  by  the  utility  and  review  by  the  NRC  staff  of  an 
application  for  a construction  permit.  The  second  stage  consists  of  the 
filing  by  the  utility  and  review  by  the  staff  of  an  application  for  an 
operating  license.  A substantive  part  of  the  two  applications  pertains  to 
quality  assurance  program  requirements  described  in  Title  10  of  the  CFR, 

Part  50,  Appendix  B (Key  Document  F-l) . These  requirements  were  published 
in  July  1970  and  define  the  elements  of  a quality  assurance  program  derived 
from  military  programs.  Detailed  descriptions  of  program  elements  and 
requirements  are  given  in  NRC's  standard  review  plans,  "Quality  Assurance 
During  the  Design  and  Construction"  (Key  Document  F-2)  and  "Quality 
Assurance  During  the  Operations  Phase”  (Key  Document  F-3) . 

NRC  also  evaluates  the  performance  of  nuclear  power  plants.  Two 
important  evaluation  (data  feedback)  tools  are  the  licensee  event  reports 
(LERs)  and  the  NRC  Gray  Book.* 

The  LER  file  contains  descriptions  of  those  plant  events  in  violation 
of  technical  specifications  primarily  in  safety-related  equipments,  furnished 
to  NRC  by  licensees.  NRC  furnishes  a bi-weekly  summary  of  LERs.  Outage  data 
are  not  normally  provided;  the  Nuclear  Safety  Information  Center  (Oak  Ridge) 


*A  comparison  of  nuclear  industry  data  bases  with  a nominal  performance 
ideal  is  given  by  M.E.  Lapides  and  Edwin  Zebroski  in  the  EPRI  report  "Use 
of  Nuclear  Plant  Operating  Experience  to  Guide  Productivity  Improvement 
Programs",  EPRI,  Palo  Alto,  California.  The  summary  offered  here  is  based 
on  the  EPRI  report. 
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periodically  republishes  the  LER  file  and  adds  topical  summaries,  distri- 
butional studies,  and  other  data,  typically  as  available  from  foreign 
sources  and  research  reactor  experience. 

The  NRC  Gray  Book  is  a monthly  summary  of  plant  performance  data,  by 
plant,  for  each  reactor  licensed  for  commercial  service.  Forced  and 
scheduled  outage  data  are  provided  by  plant  system,  subsystem,  and  component 
descriptors. 

A related  effort  is  the  Nuclear  Plant  Reliability  Data  System  (NPRDS) , 
initiated  by  EEI/ANSI  and  being  implemented  by  Southwest  Research  Institute 
based  on  data  voluntarily  provided  by  utilities.  The  system,  when  com- 
pleted, will  contain  a "pedigree  list"  (detailed  design  data)  for  plant 
safety- related  equipment  and  lists  of  "failure"  incidents  reported  against 
that  pedigree  list  by  utilities. 

Another  aspect  of  NRC's  performance  assurance  activities  is  the  devel- 
opment of  standards.  The  Office  of  Standards  Development  publishes  Regula- 
tory Guides  which  are  "issued  to  describe  and  make  available  to  the  public 
methods  acceptable  to  the  NRC  staff  of  implementing  specific  parts  of  the 
Commission's  regulations,  to  delineate  techniques  used  by  the  staff  in 
evaluating  specific  problems  or  postulated  accidents,  or  to  provide  guidance 
to  applicants  . . . compliance  is  not  required  ....  Methods  and  solutions 
different  from  those  set  out  in  the  guides  will  be  acceptable  if  they  pro- 
vide a basis  for  the  findings  requisite  to  the  issuance  or  continuance  of 
a permit  or  license  by  the  Commission".  These  guides  often  cite  and  approve 
industry  standards. 

The  guides  are  issued  in  the  following  ten  broad  divisions: 

• Power  reactors 

• Research  and  test  reactors 

• Fuels  and  materials  facilities 

• Environmental  and  siting 

• Materials  and  plant  protection 

• Products 

• Transportation 

• Occupational  health 

• Antitrust  review 

• General 


F-2 


1 


2.  APPLICABILITY  OF  NRC  PERFORMANCE  ASSURANCE  ACTIVITIES  TO  BALANCE -OF- 

PLANT  AND  NON-NUCLEAR  POWER  PLANTS 

As  pointed  out  by  Lapides  and  Zebroski,  the  most  clear-cut  incentive 
to  improve  the  performance  of  nuclear  plants  derives  from  the  cost  of 
replacement  power.  The  effect  of  a single  day's  outage  of  a 1000  MWe 
nuclear  unit  has  a current  value  of  $250,000  to  $1  million  depending  on 
local  fuel  situations.  The  dollar  magnitude,  the  impact  on  the  cost  of 
service,  and  the  necessity  of  payments  from  short-term  cash  assets  are 
the  major  sources  of  utility  incentives. 

However,  if  the  cost  of  outages  is  to  be  reduced  by  applying  perform- 
ance assurance  techniques  to  balance-of-plant  and  non-nuclear  power  plants, 
it  will  be  necessary  to  face  the  institutional  problems  related  to  the 
utilities  being  separate  and  individual  corporations  served  during  their 
design,  construction,  and  operation  by  a variety  of  independent  firms, 
contractors,  and  suppliers.  It  is  within  the  context  of  the  "institution- 
alization" problem  that  NRC's  safety-oriented  performance  assurance  programs 
offer  the  greatest  potential.  The  electric  power  industry  (at  least  those 
segments  that  have  installed  nuclear  units)  has  succeeded  in  applying  and 
using  the  methods  and  techniques  required  to  meet  NRC  regulations.  Although 
these  methods  and  techniques  (and  the  paperwork)  were  expensive  to  incor- 
porate, most  utilities  and  supporting  firms  have  succeeded  in  not  only 
incorporating  them  but  in  applying  them  efficiently  and  effectively.  NRC 
has  helped  to  simplify  them  and  reduce  the  cost  of  their  incorporation; 
for  example,  NRC  now  approves  generic,  topical  reports  so  that  the  rather 
limited  number  of  firms  that  provide  nuclear  steam  system  supplies, 
architect/engineers,  construction,  and  construction  management  services 
need  only  refer  to  the  generic,  topical  report  in  most  applications  (29 
such  reports  have  been  accepted  by  NRC) . Also,  NRC  has  approved  standard 
plant  designs  that  may  reduce  not  only  the  cost  of  meeting  regulatory 
requirements,  but  may  help  reduce  rapidly  escalating  design  and  construction 
costs.  In  short,  the  power  industry  has  already  installed  safety-oriented 
performance  assurance  programs , thereby  pacing  the  way  for  a broader 
emphasis.  It  is  conceivable  that  the  incremental  cost  of  applying  per- 
formance assurance  methods  and  techniques  to  improve  investment  effective- 
ness, reduce  life-cycle  costs,  and  improve  service  may  be  comparatively 
small  if  applied  in  conjunction  with  NRC  requirements  by  those  who  are 
experienced  in  dealing  with  NRC  requirements.  Victor  Stello,  Jr. , has 
made  some  recommendations  in  this  regard.*  Many  parts  of  NRC's  Quality 
Assurance  Program  are  applicable  to  balance-of-plant  and  non-nuclear  plants. 


*V.  Stello,  Jr. , "Some  Bases  for  a Systematic  Program  to  Enhance  Nuclear 
Plant  Reliability,"  Executive  Conference  on  Improving  Powerplant  Reliability, 
The  Homestead,  Hot  Springs,  Virginia,  27-29  September  1976. 


F-3 


3. 


DESCRIPTION  OF  APPLICABLE  PRACTICES 


3. 1  Key  Program  Elements 

NRC  requires  all  applicants  to  institute  a 18-element  quality  assurance 
(QA)  program.  These  are  summarized  below. * 

3.1.1  Organization 

• The  applicant  must  remain  responsible  for  the  establishment  and 
execution  of  the  QA  program. 

• The  applicant  may  delegate  the  work  of  establishment  and  execution. 

• The  authority  and  duties  of  persons  and  organizations  performing  QA 
functions  must  be  clearly  established  and  delineated  in  writing. 

• Persons  and  organizations  performing  QA  functions  must  be  permitted 
sufficient  authority  and  organizational  freedom  to  remain  inde- 
pendent of  cost  and  schedule  when  those  considerations  are  opposed 
to  safety  considerations. 

3.1.2  General  QA  Program  Requirements 

• The  QA  program  must  be  documented  by  written  policies,  procedures, 
or  instructions. 

• The  QA  program  must  be  carried  out  throughout  plant  life. 

• The  applicant  must  identify  the  structures,  systems,  and  components 
to  be  covered  by  the  QA  program. 

• The  applicant  must  identify  the  major  organizations  participating 
in  the  program  and  their  assigned  QA  functions. 

• Activities  affecting  quality  must  be  accommodated  under  suitably 
controlled  conditions  taking  into  account  the  need  for  special 
controls,  processes,  test  equipment,  tools,  skills,  verification  by 
inspection,  and  test. 

• The  program  must  provide  for  indoctrination  and  training  of  personnel 
performing  activities  affecting  quality. 

• The  applicant  must  regularly  review  the  status  and  adequacy  of  the 
QA  program. 


*We  have  chosen  to  follow  NRC’s  use  of  the  word  "must"  in  describing  assurance 
requirements.  By  doing  so,  we  do  not  mean  to  imply  that  envisaged  extensions 
of  these  requirements  to  balance-of-plant  and  non-nuclear  plants  should  be 
legislated.  The  word  "must"  can  be  interpreted  in  a contractual  sense. 
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3.1.3  Design  Control 

• Measures  must  be  established: 

• • To  assure  that  regulatory  requirements  are  correctly  translated 
into  specifications,  drawings,  procedures,  and  instructions 

••  To  assure  that  appropriate  standards  are  specified  and  included 
in  design  documents 

••  For  selecting  and  reviewing  the  suitability  of  materials,  parts, 
equipment,  and  processes 

• • For  identifying  and  controlling  design  interfaces  and  for  coordi- 
nating among  design  organizations 

• Design  control  measures  must  provide  for  verifying  the  adequacy  of 
design  as  by: 

••  Performing  design  reviews 

*♦  Using  alternate  calculational  methods 

• • Performing  a suitable  test  program 

• Verification  must  be  performed  by  individuals  or  groups  other  than 
the  original  designers. 

• Tests  to  verify  design  adequacy,  used  in  lieu  of  other  verification 
processes,  must  include  qualifications  testing  of  a prototype  under 
the  most  adverse  design  conditions. 

• Design  changes  must  be  subjected  to  design  control  measures  com- 
mensurate with  those  applied  to  the  original  design  and  be  approved 
by  the  original  design  organization  or  other  independent  designee. 

3.1.4  Procurement  Document  Control 

• Applicable  regulatory  requirements,  design  bases,  and  other  QA 
requirements  must  be  cited  in  procurement  documents. 

• Procurement  documents  must  require  contractors  and  subcontractors 
to  provide  an  appropriate  quality  assurance  program. 

3.1.5  Instructions,  Procedures,  and  Drawings 

• Activities  affecting  quality  must  be  prescribed  and  accomplished  in 
accordance  with  documented  instructions,  procedures,  or  drawings. 

• Instructions,  procedures,  and  drawings  must  include  appropriate 
acceptance  criteria. 

3.1.6  Document  Control 

Measures  must  be  established  to  control  the  issuance  of  documents 
which  prescribe  QA  activities  and  assure  that  they  are  reviewed  for  adequacy. 
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3.1.7  Control  of  Purchased  Material,  Equipment,  and  Services 

• Measures  msut  be  established  to  assure  that  purchases  conform  to 
procurement  documents  and  quality  requirements. 

• Documentary  evidence  that  material  and  equipment  conform  to  pro- 
curement requirements  must  be  available  at  the  nuclear  power  plant 
site. 

3.1.8  Identification  and  Control  of  Materials,  Parts,  and  Components 

Identification  and  control  measures  must  be  designed  to  prevent  the 
use  of  incorrect  or  defective  material,  parts,  or  components. 

3.1.9  Control  of  Special  Processes 

Measures  must  be  established  to  assure  that  special  processes  including 
welding,  heat  treating,  and  nondestructive  testing  are  controlled  and 
accomplished: 

• By  qualified  personnel 

• Using  qualified  procedures 

• In  accordance  with  applicable  codes,  standards,  specifications, 
criteria,  and  other  special  requirements 

3.1.10  Inspection 

• A program  for  inspection  of  process  monitoring  of  activities  affect- 
ing quality  must  be  established. 

• Inspections  must  be  performed  by  individuals  other  than  those  who 
performed  the  activity. 

• Both  inspection  and  process  monitoring  must  be  provided  when  control 
is  inadequate  without  both. 

• Mandatory  inspections  without  which  work  cannot  proceed  must  be 
identified  in  appropriate  documents. 

3.1.11  Test  Control 

• A program  must  be  established  to  assure  that  all  testing  required 
to  demonstrate  satisfactory  in-servict.  performance  is  identified 
and  performed  in  accordance  with  written  test  procedures. 

• The  test  program  must  include: 

• • Proof  tests  prior  to  installation 
• • Preoperational  tests 
• • Operational  tests 

• Test  results  must  be  documented  and  evaluated  to  assur*  • •• 

requirements  have  been  satisfied. 
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3.1.12  Control  of  Measuring  and  Test  Equipment 


Measures  must  be  established  to  assure  that  tools,  gages,  instruments, 
and  other  measuring  and  testing  devices  are  properly  controlled,  calibrated, 
and  adjusted  at  specified  periods. 

3.1.13  Handling,  Storage,  and  Shipping 

• Measures  must  be  established  to  control  handling,  storage,  shipping, 
cleaning,  and  preservation  of  material  and  equipment  in  accordance 
with  work  and  inspection  instructions. 

• Special  protective  environments  must  be  specified  and  provided  if 
necessary. 

3.1.14  Inspection,  Test,  and  Operating  Status 

• Measures  must  be  established  to  mark  the  status  of  inspections  and 
tests  performed  upon  individual  items. 

• Measures  must  be  established  for  indicating  the  operating  status  of 
structures,  systems,  and  components  to  prevent  inadvertent  operation. 

3.1.15  Nonconforming  Materials , Parts,  or  Components 

• Measures  must  be  established  to  control  materials,  parts,  or  com- 
ponents which  do  not  conform  to  requirements  to  prevent  inadvertent 
use. 

• Nonconforming  items  must  be  reviewed  and  accepted,  rejected,  repaired, 
or  reworked  in  accordance  with  documented  procedures. 

3.1.16  Corrective  Action 


• Measures  must  be  established  to  assure  that  failures,  malfunctions, 
deficiencies,  deviations,  defectives,  and  nonconformances  are 
promptly  identified  and  corrected. 

• In  the  case  of  significant  conditions  adverse  to  quality,  the  cause 
of  the  condition  must  be  determined. 

• The  cause  of  the  condition  and  the  corrective  action  taken  must  be 
documented  and  reported. 

3.1.17  Quality  Assurance  Records 

• Operating  logs  and  the  results  of  reviews,  inspections,  tests,  audits, 
monitoring  of  work  performance,  and  materials  analysis  must  be 
recorded  and  maintained. 

• Records  must  also  include  qualifications  of  personnel,  procedures, 
and  equipment. 
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• Inspection  and  test  records  must  identify  the  inspector  or  data 
recorder,  the  type  of  observation,  the  results,  the  acceptability, 
and  the  action  taken  in  connection  with  deficiencies  noted. 

• The  applicant  must  establish  requirements  concerning  record  retention, 
such  as  duration,  location,  and  assigned  responsibility. 

3.1.18  Audits 


• A comprehensive  system  of  planned  and  periodic  audits  must  be  carried 
out  to  verify  compliance  and  determine  the  effectiveness  of  the 
quality  assurance  program. 

• Audits  must  be  performed  in  accordance  with  written  procedures  or 
check  lists. 

• Audit  personnel  must  be  appropriately  trained. 

• Audio  personnel  must  not  have  direct  responsibility  in  the  areas  being 
audited. 

• Audit  results  must  be  documented  and  reviewed  by  those  responsible 
in  the  audited  area. 

• Follow-up  action  including  reaudit  of  deficient  areas  is  required. 

3. 2 Lead  Responsibilities  and  Government- Industry  Interface 

Although  NRC  provides  the  impetus  for  applying  the  quality  assurance 
program  and  has  the  ultimate  responsibility  for  approving  the  quality 
assurance  plan  and  the  result,  the  licensee  is  responsible  for  applying  the 
program.  Typically,  the  licensee  delegates  a considerable  portion  of  the 
implementation  responsibility  to  specialized  contractors,  thereby  creating 
a vertically  integrated  situation  held  together  by  legislated  regulations 
at  the  top  and  contractual  requirements  at  the  bottom. 


4.  COST  AND  EFFECTIVENESS 

NRC 1 s annual  operating  budget  is  about  $281  million. 

It  is  difficult  to  objectively  judge  the  effectiveness  of  NRC's  pro- 
gram because  numerical  performance  goals  such  as  mean  time  between  failures 
(MTBF)  and  mean  time  to  report  (MTTR)  are  not  specified  and  allocated.  It 
is  interesting  to  note  that  NRC  has  sponsored  one  of  the  most  comprehensive 
and  well-documented  studies  of  quantitative  risk  ever  assembled,  but  has  not 
converted  the  results  into  goals. 
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AMERICAN  TELEPHONE  AND  TELEGRAPH  (AT&T) 


1 . HISTORICAL  DEVELOPMENT 

Performance  assurance  policies,  procedures,  and  practices  are  inter- 
woven into  the  fabric  of  the  Bell  System*  organization  structure  in  such 
an  intimate  way  that  it  is  difficult  to  separate  the  performance  assurance 
functions  from  any  other.  The  organizational  structure  and  capital  plant 
have  grown  together  over  many  years.  In  fact.  Bell's  performance  assurance 
program  is  the  most  highly  evolved  of  any  discussed  in  this  report.  Be- 
cause it  has  been  relieved  from  the  pressure  of  external  competition,  Bell 
System  management  has  been  able  to  introduce  new  technology  at  an  incremen- 
tal rate  and  to  continually  develop,  refine,  test,  and  formalize  an  organi- 
zational structure  to  cope  with  and  utilize  new  technology  very  effectively. 

2.  SCOPE  OF  PERFORMANCE  ASSURANCE  PRACTICES 

Every  step  in  the  research,  development,  manufacture,  installation, 
testing,  and  operation  process  is  coordinated  and  integrated  by  AT&T  policy 
guidelines,  organizational  charters,  and  standard  practices  which  are  used 
throughout  the  Bell  System.  The  results  is  a sophisticated,  vertically 
integrated,  closed- loop  performance  assurance  program  of  unrivaled  scope 
and  depth. 

AT&T**  is  the  headquarters  organization,  which  coordinates  the  entire 
enterprise  in  five  ways: 

• By  stock  ownership  in  the  operating  companies 

• By  functioning  as  a general  staff  to  assist  the  23  operating 
companies 

• By  furnishing,  through  its  Long  Lines  Department,  interstate 
service  between  the  different  operating  companies 

*The  word  "system"  as  used  here  follows  AT&T's  use  of  the  word  to  denote 
the  combination  of  organization  and  plant  required  to  deliver  telecom- 
munication service. 

**AT&T  consists  of  the  general  department  (designated  "195")  and  the  Long 
Lines  Department. 
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• By  ownership  of  the  Western  Electric  Company,  the  manufacturing 
and  supply  unit  of  the  Bell  System 

• By  ownership,  with  Western  Electric  (50-50) , of  the  Bell  Telephone 
Laboratories  (BTL) , which  performs  research,  development , and 
testing  work  to  improve  capital  plant 

BTL  had  16,000  employees  in  1975.  It  is  funded  by  the  operating  com- 
panies and  the  Long  Lines  Department  for  research  and  fundamental  develop- 
ment ($227  million  to  $250  million  in  1975) , by  Western  Electric  for  spe- 
cific development,  design,  and  testing  ($321  million  in  1975) , by  the 
operating  companies  for  information  systems  ($35  million  to  $65  million 
in  1975),  and  by  the  U.S.  Department  of  Defense  ($53  million  in  1975). 
Usually,  BTL  has  spent  roughly  equal  amounts  on  electronics,  transmission , 
and  switching,  equally  divided  between  research  and  customer  products. 


Western  Electric,  with  153,000  employees,  spends  about  60  percent  of 
its  annual  budget  on  manufacturing,  about  33  percent  on  services  (mainly 
installation) , and  about  7 percent  on  purchasing  and  transportation.  Sales 
in  1975  were  $6,127  billion  of  which  93  million  were  to  the  Bell  System*. 

Both  BTL  and  Western  Electric  play  key  roles  in  establishing  perform- 
ance assurance  goals,  defining  performance  assurance  requirements,  and 
implementing  performance  assurance  programs. 


3.  APPLICABILITY  OF  BELL  SYSTEM  PRACTICES  TO  THE  ELECTRIC  POWER  INDUSTRY 

The  aspects  of  the  Bell  System's  approach  to  performance  assurance 
which  are  most  unusual  are  the  organizational  aspects.  Bell  has  organized 
performance  assurance  on  a national  scale.  The  implementation  structure 
is  vertically  integrated  in  a manner  and  to  an  extent  that  may  not  be  ap- 
plicable in  the  electric  power  industry,  but  the  Bell  System  organ izational 
precedent  is  worth  examining  if  only  to  show  how  a national-scale  perform- 
ance assurance  program  could  make  use  of  centralized  and  shared  skill  banks 
(like  BTL's);  coordinated  manufacturing,  installation,  and  repair  services 
(like  Western  Electric's);  and  vertically  integrated  maintenance  on  a nation- 
al scale  (like  that  of  the  operating  companies,  Western  Electric,  and  BTL) . 

Also,  the  Bell  System  displays  a unique  ability  to  manage  technological 
change.  Bell  does  so  by  requiring  extensive  field  trials  of  prototype 
equipment,  before  making  new  equipment  available  for  general  purchase,  de- 
ploying specialized  technical  teams  to  respond  to  new  equipment  problems, 
collecting  and  analyzing  performance  data  in  three  tiers,  standardizing 
equipment  at  the  major  component  level  in  the  system,  and  continually  ana- 
lyzing optimum  levels  of  redundancy  at  local  and  national  levels.  These 
methods  of  managing  technical  change  should  be  considered  by  the  Department 
of  Energy  (DOE)  and  the  power  industry  to  the  extent  that  anti-trust 
legislation  allows. 

•Recently,  Western  has  begun  selling  complete  telephone  systems  to  under- 
developed countries. 
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4.  DESCRIPTION  OF  APPLICABLE  ORGANIZATION  AND  PRACTICES 

4.1  Bell  Telephone  Laboratories'  Role 

As  a general  function  performance  assurance  task,  BTL  controls  the 
rate  at  which  new  technology  is  introduced  into  the  system.  BTL  performs 
this  function  during  the  research  and  fundamental  development  phases  by 
assessing,  for  cost-effective  application  to  the  Bell  System,  new  techno- 
logical developments  derived  from  outside  and  inside  the  Bell  System.  The 
tendency  is  to  open  the  new  technology  "filter"  as  wide  as  possible  during 
the  research  phase  and  reduce  the  filter  aperture  during  the  fundamental 
development  phase  by  applying  BTL's  intimate  and  sophisticated  knowledge  of 
deployment  and  operating  constraints  — both  economic  and  technical.  Dur- 
ing the  specific  development  phase,  BTL  concentrates  on  proving  that  the 
technology  can  be  applied  as  predicted.  BTL  practices  the  "informed  deci- 
sion" approach  to  the  process  of  applying  new  technology  with  primary 
emphasis  on  the  "informed"  part  of  the  phrase.  BTL  proceeds  slowly,  allow- 
ing plenty  of  time  to  cope  with  infant  mortality  problems,  recognizing 
that  economies  of  scale  can  only  be  realized  when  the  technology  reaches 
maturity. 

Typically,  the  associated  companies  request  that  BTL  design  a new 
item.  BTL  is  funded  for  an  initial  study  during  which  corporate-wide  re- 
quirements with  respect  to  cost  and  compatibility  are  investigated  thorough- 
ly. Availability,  reliability,  and  maintainability  requirements  are  de- 
rived from  cost  and  compatibility  considerations.  BTL  then  designs  and 
builds  prototypes  for  field  trial  and,  in  the  process,  designs  appropriate 
test  equipment  and  procedures,  diagnostic  techniques,  redundancy  rules,  and 
technical  specifications.  BTL  also  provides  a detailed  maintenance  plan 
that  is  refined  as  the  system  is  developed  and  field-tested. 

4.2  Western  Electric's  Role 

Western  Electric  plays  a major  role,  with  BTL,  in  addressing  installa- 
tion and  logistical  support  requirements  during  the  specific  development 
phase.  Western  Electric  manufactures  the  items  to  BTL's  specifications, 
develops  engineering  and  operating  standards,  and  publishes  appropriate 
Bell  System  Practices  (e.g. , test  standards  that  define  how  the  item  should 
be  tested  on  a stand-alone  and  installed  basis).  One  of  Western  Electric's 
major  roles  is  quality  assurance.  Typically,  Western  Electric  places  more 
emphasis  on  screening  and  qualification  testing  than  would  be  the  case  if 
there  were  competitive  pressures  to  get  into  production. 


4.3  The  Role  of  the  Operating  Companies 

Because  the  operating  companies  have  direct  interfaces  with  customers, 
there  is  strong  emphasis  on  immediate  response  to  service  calls.  Each  com- 
pany maintains  highly  trained  and,  typically,  very  experienced  diagnostic 


and  repair  crews.  These  crews  are  members  of  a vertically  integrated  group 
of  engineering  personnel  who  play  central  roles  at  each  level  in  the  entire 
Bell  System  organization.  They  pass  along  problems  and  solutions  as 
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necessary.  In  essence,  the  maintenance  and  repair  functions  are  performed 
within  a specialized  vertical  organization  that  is  nominally  separable  from 
and  parallel  to  the  main  structure.  Thus,  all  operating  companies  and  each 
customer  is  able  to  utilize  the  services  of  top  echelon  personnel  at  BTL, 
Western  Electric,  and  Long  Lines,  if  necessary.  Specialization  increases 
from  the  top  down,  whereas  experience  and  education  increase  from  the  bottom 
up. 


4.4  Field  Trials 

The  Bell  System  employs  acceptance  and  demonstration  testing  less 
frequently  than  is  typical  when  new  items  are  acquired  and  operated  by 
separate  organizations . Instead,  Bell  relies  on  extensive  field  trials 
performed  by  a team  comprising  BTL,  Western,  195,  and  one  or  more  operating 
companies  (and/or  Long  Lines) . Many  of  the  trials  last  months  and  some 
last  years.  They  are  used  to  work  out  bugs,  demonstrate  reliability  and 
maintainability,  assess  economic  and  organizational  impact,  and  prepare 
operating  plans,  procedures,  and  standard  practices. 

4.5  The  Technical  Centers  Concept 

When  a new  technology  is  introduced,  Bell  augments  the  maintenance 
and  repair  structure  described  above  by  training  and  deploying  special 
teams  organized  in  "centers".  These  centers  respond  to  maintenance  and 
repair  requests  from  any  operating  company.  The  first  of  such  centers  was 
a group  of  Data  Technical  Assistance  Centers  (DATAC)  organized  to  deal  with 
data  communications  problems.  DATAC  was  so  effective  that  Bell  is  training 
and  will  soon  deploy  other  center  teams  such  as  SCOTS  (Surveillance  and 
Control  of  Transmission  Systems) , ESSTAC  (Electronic  Switching  System  Tech- 
nical Assistance  Centers),  the  Centralized  Data  Testing  Centers,  and  the 
Facility  Maintenance  Management  Centers. 

4.6  Performance  Data  Systems 

After  an  item  has  become  available  on  a limited  or  general  basis  to 
the  operating  companies,  performance  data  is  fed  back  in  three  formal  ways. 

One  way  is  the  Engineering  Complaint  Process.  If  the  item  is  available 
on  a limited  basis,  such  as  during  field  trials,  performance  exceptions  are 
submitted  to  the  original  development  group  or  field  trial  team.  If  the 
item  is  generally  available,  performance  exceptions  are  submitted  to  the 
Department  Engineer,  the  Western  Electric  Field  Representative,  and  the  BTL 
Field  Representative. 

A second  way  to  feed  back  performance  data  is  through  the  Trouble 
Ticket  and  Trouble  Coding  System.  Failures,  outages,  repair  actions,  and 
causes  are  sent  to  195.  A computerized  reporting  system  maintained  by 
195  performs  many  types  of  analysis , J including  cross-departmental  .nalysis. 
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The  third  formal  feedback  mechanism  is  through  the  Routine  Maintenance 
Reporting  System.  BTL  prescribes  routine  maintenance  and  publishes  mainte- 
nance practices  and  Task  Oriented  Plant  Practices  (TOPPS)  derived  and  up- 
dated, in  part,  from  reviewing  routine  maintenance  reports. 

4.7  Standardization 

The  Bell  System  is  one  of  the  few  organizations  that  standardize  at 
the  module  level.  Although  Bell  recognizes  the  inherent  dangers  of  equip- 
ment standardization  above  the  piece-part  level,  it  also  recognizes  that 
the  risks  of  allowing  rapid  technological  change  increase  rapidly  with  sys- 
tem scale.  Furthermore,  BTL  continually  assesses  the  impact  of  large-scale 
standardization  by  means  of  sophisticated  models  that  consider  all  pertinent 
factors  and,  because  the  Bell  System  controls  all  aspects  of  applicable 
technological  growth,  it  is  possible  to  tailor  the  standardization  policy 
accordingly . 

4.8  System  Redundancy  on  a Local  and  National  Scale 

The  Bell  System  has  taken  advantage  of  national  and  local  redundancy 
allocation  for  many  years. 

National-level  redundancy  patterns  are  planned  and  implemented  by  the 
Long  Lines  Department.  This  department  has  total  responsibility  for  coor- 
dinating regional  and  national  (and  international)  interconnection  require- 
ments. More  significantly.  Long  Lines  also  has  separate  and  distinct 
budgetary  authority  as  well  as  separate  and  distinct  profit  and  loss  re- 
sponsibilities (i.e. , through  the  "toll"  approach).  Therefore,  Long  Lines 
can  and  does  act  as  an  advocate  for  national-scale  redundancy  and  cam  back 
up  its  position  rather  effectively  in  opposition  to  advocates  of  local 
redundancy,  if  required.  Although  the  cost-effectiveness  of  national  vs. 
local  redundancy  patterns  is  controversial.  Bell  people  believe  that  it  is 
important  that  advocates  of  national-level  redundancy  have  at  least  as  much 
economic  and  technical  power  as  local  advocates  if  cost-effective  compro- 
mises are  to  evolve  in  time  to  cope  with  "high-cost-of-local-redundancy" 
problems  as  they  occur. 

Local-level  redundancy  patterns  typically  evolve  in  response  to  the 
peculiar  characteristics  of  local  demand.  Since  the  early  1930s,  the 
principal  determinants  have  been — first, the  trend  toward  urbanization  and, 
subsequently,  the  trend  toward  urban  sprawl.  During  the  sprawl  stage  the 
need  for  more  intra-regional  redundancy  became  evident  in  some  areas  of  the 
country  (e.g.,  the  Northeast  Corridor)  at  about  the  same  time  as  needs  for 
intra-national  redundancy  became  apparent.  As  one  result,  the  former  em- 
phasis on  local  (i.e.,  state-by-state)  regulation  became  a deterent  to 
cost-effective,  "integrated"  redundancy  at  all  levels  (e.g.,  the  Nevada 
"rusty  switch"). 

Both  viewpoints,  the  local  and  national,  are  centrally  addressed  by 
BTL.  In  performing  this  function,  BTL  provides  inputs  to  both  the  long 
lines  and  local  planning  processes.  Furthermore,  BTL  is  able  to  act  as 
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referee  in  the  bi-advocate  process  by  providing  sophisticated  analytic  help 
to  both  parties.  BTL  considers  economic  and  technical  alternatives  at  all 
levels. 


5.  COST  AND  EFFECTIVENESS 

The  Bell  performance  assurance  program  has  survived  and  evolved  over 
many  years  in  a very  efficient  and  highly  structured  environment,  and  ap- 
pears to  be  cost-effective  in  that  environment.  However,  we  were  not  able 
to  obtain  enough  explicit  and  quantitative  information  to  provide  an 
objective  measure. 

5.1  Cost  of  Performance  Assurance  at  Bell 

Bell  spends  a substantial  percentage  of  its  budget  for  new  systems  on 
performance  assurance.  When  one  considers  the  reports  published  by  BTL  and 
the  fact  that  many  prototypes  undergo  extensive  field  trial,  it  appears 
that  performance  assurance  costs  at  Bell  may  be  very  high  relative  to  pro- 
totype acquisition  costs  for  complex  systems  (e.g.,  electronic  switches). 
However,  once  prototype  performance  has  been  assured,  the  system  is  stan- 
dardized and  manufactured  on  a semi-mass-production  basis.  Furthermore, 
the  production  systems  are  designed  for  very  long  life  and  depreciated 
over  many  years.  Thus,  the  cost  of  performance  assurance  relative  to  the 
cost  of  total  plant  is  probably  very  small. 

5.2  Effectiveness  of  Bell's  Performance  Assurance  Program 

The  Bell  System's  low  cost  to  consumers  and  dependable  service  is 
evidence  that  Bell's  program  is  reasonably  effective  from  a life-cycle-cost 
viewpoint.  However,  there  is  considerable  controversy  whether  the  Bell 
approach  maximizes  availability,  reliability,  and  maintainability  and 
minimizes  redundancy  in  an  optimum  way. 

The  Bell  System  program  appears  to  be  very  effective  in  uniting  and 
focusing  national  as  well  as  local  efforts.  Furthermore,  most  of  the  high 
risks  are  assumed  at  the  nonlocal  level  (by  AT&T  through  BTL) . These 
risks  are  allocated  equally  to  all  local  levels  and  spread  over  long  periods 
of  time  (via  pricing  policies  which  explicitly  recognize  new  system 
development  costs) . 
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